Main grid unmanned plane imitative line flight path correction method and system, computer device and medium

Abstract

The application relates to the technical field of power line inspection, in particular to a main distribution network unmanned aerial vehicle (UAV) line simulation flight path correction method and system, computer equipment and a medium; the method comprises the following steps: collecting real-time sensing data of the UAV to generate a state observation vector; outputting a high-level control instruction to a bottom-layer flight control system for execution; in response to UAV state migration, storing experience data into an experience playback buffer; updating network parameters of a reinforcement learning intelligent agent; during flight task execution, cyclically executing the steps of state observation vector generation, instruction issuing, safety monitoring and experience storage, and executing the step of network parameter updating; in this way, the technical problem that in the prior art, path tracking accuracy, environmental adaptability and flight safety are difficult to be cooperatively guaranteed in a complex line corridor environment is solved, and the autonomy, reliability and overall efficiency of the UAV in the power line inspection task are improved.

Description

Technical Field

[0001] This invention relates to the field of power line inspection technology, and in particular to a method, system, computer equipment, and medium for correcting the flight path of a main distribution network drone following a power line. Background Technology

[0002] With the deepening of the intelligent transformation of the power system, the scale and precision requirements of the inspection and maintenance of main and distribution network lines are continuously increasing. Due to their maneuverability, flexibility, and high efficiency, drones have been gradually applied in the field of line inspection, replacing traditional manual methods to cope with the increasing workload. In complex geographical environments such as mountainous and forested areas, drones need to achieve high-precision line-following flight to complete tasks such as defect detection and corridor inspection, which poses a severe challenge to the stable tracking and real-time correction capabilities of the flight path.

[0003] In the field of UAV line-following flight and path correction, several typical solutions have been developed. Tracking based on preset waypoints using the Global Positioning System (GPS) has long been widely used due to its simplicity and ease of implementation. However, in complex environments where signals are easily blocked, this method struggles to guarantee positioning accuracy and continuity, and it cannot perceive spatial morphological changes caused by the sag of the route itself or wind deflection, leading to tracking deviations. On the other hand, visual navigation-based line recognition and tracking methods attempt to directly perceive the environment through image processing, but their performance is easily affected by lighting, weather, and background interference, lacking the ability to predict the direction of the route ahead and potential obstacles, and prone to tracking loss or control oscillations in dynamic environments.

[0004] To improve environmental adaptability, some studies have begun to introduce more advanced adaptive control methods. Existing technology (application publication number CN120143870A) discloses an adaptive control method and system for UAV flight trajectory correction. This method constructs an environmental model by fusing multi-sensor data and performs trajectory prediction and three-axis compensation based on point cloud matching and pose estimation to achieve flight correction in complex environments. This method represents a technological direction of improving adaptability through environmental perception and model prediction. However, when facing specific route corridor inspection scenarios with long distances, continuous changes, and numerous rigid obstacles (such as towers and conductors), this type of method still faces significant limitations. First, it highly relies on real-time and accurate modeling and matching of the flight environment. When the inspection route background is complex, there is interference from similar features, or sensors are affected by weather noise, the reliability of the model and the accuracy of pose calculation may decrease, affecting the accuracy of correction. Second, the core of this method lies in compensating for deviations in the preset flight path. Its decision-making logic is relatively fixed, making it difficult to accumulate and utilize historical flight experience. When facing complex disturbances such as sudden changes in route direction and continuous crosswinds, the flexibility and adaptive adjustment capability of the strategy may be insufficient. Furthermore, while the calibration process considers multi-axis adjustment, the safety mechanisms are primarily embedded within the control loop. It lacks depth in addressing independent and robust safety takeover and switching mechanisms for situations such as control command oscillations, actuator saturation, or sudden abnormalities (e.g., sensor malfunctions or strong airflow impacts) that may occur under complex strategies. This makes it difficult to meet the stringent requirements for absolute flight safety in line inspections. Therefore, existing UAV line-following flight path correction technologies suffer from the technical challenge of coordinating path tracking accuracy, environmental adaptability, and flight safety in complex line corridor environments. Summary of the Invention

[0005] To address the aforementioned shortcomings or deficiencies, this invention provides a method, system, computer equipment, and medium for correcting the flight path of a main power grid UAV following a line. This solution addresses the technical problem that existing technologies struggle to coordinate path tracking accuracy, environmental adaptability, and flight safety in complex line corridor environments.

[0006] This invention provides a method for correcting the flight path of a UAV using a main-distribution network, comprising: collecting real-time sensor data of the UAV and comparing it with a preset reference path model to generate a state observation vector; inputting the state observation vector into the policy network of a reinforcement learning agent to output higher-level control commands; issuing the higher-level control commands to the lower-level flight control system for execution, while an independent safety monitoring module monitors the flight status in real time and triggers a switch of control when the state exceeds the limit; responding to the UAV state transition, collecting the new environmental state and calculating the reward according to the reward function, and storing the experience data including the state, action, reward, and next state into an experience replay buffer; sampling historical experience data from the experience replay buffer to update the network parameters of the reinforcement learning agent; and cyclically executing the steps of generating the state observation vector, issuing commands and conducting safety monitoring, and storing experience data during the flight mission, while asynchronously executing the step of updating the network parameters.

[0007] According to a second aspect, the present invention provides a main power grid unmanned aerial vehicle (UAV) flight path correction system, comprising: The observation vector generation module is used to collect real-time sensor data from the UAV and compare it with a preset reference path model to generate state observation vectors.

[0008] The control command output module is used to input the state observation vector into the policy network of the reinforcement learning agent and output high-level control commands.

[0009] The control switching module is used to send high-level control commands to the low-level flight control system for execution. At the same time, an independent safety monitoring module monitors the flight status in real time and triggers control switching when the status exceeds the limit.

[0010] The experience data caching module is used to collect new environmental states and calculate rewards based on reward functions when the UAV changes state. It stores the experience data, including state, action, reward and next state, into the experience replay buffer.

[0011] The network parameter update module is used to sample historical experience data from the experience replay buffer and update the network parameters of the reinforcement learning agent.

[0012] The flight path correction module is used to cyclically execute the steps of generating state observation vectors, issuing commands and monitoring safety, and storing experience during the flight mission, and to asynchronously execute the step of updating network parameters.

[0013] According to a third aspect, the present invention provides a computer device comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform any of the main distribution network UAV line-following flight path correction methods in the embodiments of the present invention.

[0014] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute any of the main and distribution network UAV line-following flight path correction methods in the embodiments of the present invention.

[0015] The present invention provides a method for correcting the flight path of a main distribution network UAV by means of line simulation. This method is achieved through five core steps: state observation, intelligent decision-making, safety monitoring, experience learning and asynchronous optimization. The system comprises several key components: Real-time sensor data from the UAV is collected and compared with a pre-defined reference path model to generate state observation vectors, enabling real-time perception and quantitative representation of the UAV's current pose and environmental characteristics; these state observation vectors are input into the policy network of the reinforcement learning agent, which outputs high-level control commands to achieve intelligent adaptive decision-making based on environmental interaction; high-level control commands are then sent to the underlying flight control system for execution, while an independent safety monitoring module monitors the flight status in real time and triggers a switch of control when the state exceeds limits, ensuring basic flight safety while introducing intelligent decision-making; in response to UAV state transitions, new environmental states are collected, and rewards are calculated based on a reward function; experience data including state, action, reward, and the next state is stored in an experience replay buffer for structured recording and storage of decision-making process data; historical experience data is sampled from the experience replay buffer and the network parameters of the reinforcement learning agent are updated for continuous iterative optimization of the decision model; during flight mission execution, the generation of state observation vectors, command issuance and safety monitoring, and experience storage steps are executed cyclically, with network parameter updates performed asynchronously to achieve parallel processing and non-interference between flight control and model learning.

[0016] In this technical solution, the present invention addresses the problem of "insufficient path tracking accuracy" mentioned in the background technology by generating state observation vectors and comparing them with a reference model in real time, providing precise quantitative input for control decisions and establishing a data foundation for high-precision tracking. Regarding the problem of "insufficient environmental adaptability and decision-making flexibility," the invention uses a reinforcement learning agent to output high-level control commands and asynchronously updates network parameters based on a reward function and experience replay mechanism. This allows the decision-making strategy to self-optimize online using historical flight experience, adapting to complex disturbances such as sudden changes in route direction and crosswinds, overcoming the technical limitations of traditional fixed-parameter or model-dependent methods with limited adaptive capabilities. Finally, regarding the problem of "imperfect flight safety assurance mechanisms," the invention constructs a safety redundancy barrier independent of the learning algorithm by setting up a safety monitoring module independent of the decision loop and forcibly switching control when the state exceeds the limit. This effectively avoids flight risks caused by agent exploration or decision-making errors, solving the defects of deep coupling between safety mechanisms and decision logic and insufficient reliability in existing technologies. Therefore, the technical solution of the present invention solves the technical problem that existing technologies have difficulty in coordinating path tracking accuracy, environmental adaptability and flight safety in complex power line corridor environments, and improves the autonomy, reliability and overall efficiency of UAVs in power line inspection tasks. Attached Figure Description

[0017] Figure 1 This is a flowchart of a method for correcting the flight path of a main power distribution network UAV according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the adaptive path tracking control process in another embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of a main power grid UAV flight path correction system according to an embodiment of the present invention; Figure 4 This is a block diagram of a computer device for implementing embodiments of the present invention. Detailed Implementation

[0018] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] During the development of this invention, the inventors, through extensive experiments and data analysis, revealed the intrinsic connection between online exploration and optimization by reinforcement learning agents and the absolute safety that aircraft must ensure in complex dynamic environments: unconstrained online policy exploration directly on a physical platform can lead to unacceptable flight risks, while relying entirely on pre-set fixed rules or models cannot adapt to unforeseen complex environmental changes; these two constitute a core contradiction. Based on this relationship, the inventors innovatively proposed this technical solution, utilizing an asynchronous parallel computing architecture. By logically and temporally decoupling the real-time perception decision loop from the offline policy learning loop, and combining this with the introduction of an independent runtime safety monitoring mechanism with the highest priority in the decision loop, this solution releases the agent's ability to autonomously optimize and adapt to the environment while ensuring the hard constraints of flight safety, embodying the core concept of "online adaptive learning under safety constraints."

[0020] Specifically, through comparative experiments, the invention team discovered that traditional control methods based on fixed models or preset rules suffer from poor environmental adaptability and an inability to learn from historical flight experience. Their control parameters and logic cannot dynamically adjust themselves based on unmodeled changes in route morphology, sudden wind disturbances, and other unforeseen events, leading to decreased tracking accuracy or even target loss in complex corridors. These technical deficiencies limit the reliability and intelligence level of inspection operations. The method proposed in this invention, which generates state observation vectors based on the comparison of real-time sensor data and reference path models, can improve the accuracy of environmental perception and state representation, providing reliable input for intelligent decision-making. By inputting the state observation vectors into the policy network of the reinforcement learning agent to output high-level control commands, the control strategy can be autonomously generated and continuously optimized based on environmental feedback, breaking through the limitations of fixed rules. By setting up an independent safety monitoring module and triggering control switching when the state exceeds the boundary, it can be ensured that the flight behavior does not exceed the preset safety boundary under any circumstances, providing a "safety barrier" for online learning. By storing the experience data generated by the decision in a buffer and asynchronously sampling to update the network parameters, and combining the cyclically executed control loop with the asynchronously executed learning steps, it can be ensured that the optimization process of the strategy does not interfere with the deterministic and time-delay requirements of high-frequency real-time control, and finally output a continuous and stable linear flight control command sequence with high precision, strong adaptability and high safety.

[0021] Therefore, this invention provides a method for correcting the flight path of a UAV tracing a main power distribution network, based on the first aspect. This method can be applied to a real-time correction system for the flight path of a UAV tracing a main power distribution network (hereinafter referred to as the "system"). This system can run on a UAV's onboard flight control computer, a ground control station server, or a cloud computing platform via software programs or firmware to complete autonomous tracing flight inspection tasks of the main power distribution network lines. Specifically, this system can be deployed in various hardware environments, including but not limited to: UAV-borne embedded computing units equipped with high-performance computing modules, industrial computers within ground mobile or fixed control stations, and virtualized server clusters in the cloud. This flexible deployment architecture allows the system to meet both the low latency and high reliability requirements of online real-time flight control and safety monitoring, and the high computing power and scalability requirements of offline model training, big data analysis, and remote task planning.

[0022] like Figure 1 As shown, the method may include: Step S110: Collect real-time sensor data from the UAV and compare it with a preset reference path model to generate a state observation vector.

[0023] Real-time sensing data refers to the multi-dimensional measurement data stream reflecting the UAV's own state and surrounding environment, acquired in real time by onboard sensors during flight. The reference path model is a pre-constructed digital three-dimensional path description based on the precise spatial coordinates, geometric shape, and obstacle information along the target inspection route (such as a high-voltage transmission line), used to provide the UAV with an ideal flight trajectory reference. The state observation vector is a multi-dimensional vector that quantifies the UAV's current state and environmental information, serving as input to the decision-making system.

[0024] Specifically, the system can acquire centimeter-level position, attitude, and environmental point cloud of the UAV by fusing data from a Real-time Kinematic (RTK) Global Navigation Satellite System (GNSS) receiver, an Inertial Measurement Unit (IMU), and a binocular vision sensor, and then register and compare them with a reference path model.

[0025] For example, the system is configured with a drone platform integrating an RTK-GNSS module, a six-axis IMU, and a 4K (4,000-pixel resolution) high-definition binocular camera. When inspecting a 220 kV transmission line, the system calculates in real time the deviations of the drone relative to the middle phase conductor in the lateral, vertical, and yaw directions as 0.15 meters (m), -0.08 meters, and 1.5 degrees (°), respectively. These deviations, the rate of change of deviation, and the curvature features of the line ahead extracted from the image are combined to generate a state observation vector containing 12 dimensions.

[0026] In some embodiments, the system can obtain the state observation vector for decision-making using the following formula (1): (1); Formula (1) defines the state observation vector at time t. It consists of four parts: positional deviation, deviation rate of change, path geometric prospective characteristics, and environmental threat prospective characteristics. This represents the three-dimensional positional deviation of the UAV relative to the reference path. These represent lateral, vertical, and heading deviations, respectively. The first derivative of the aforementioned pose deviation, i.e., the rate of change of deviation, is used to reflect the dynamic motion trend of the UAV. The geometric look-ahead features of the path. This represents the average curvature of the current line. Represents the gradient of curvature change. These characteristics represent the line deflection at the predicted point ahead, and together they characterize the curvature and changes of the path ahead. As a forward-looking feature of environmental threats, This represents the two-dimensional bounding box information of targets such as power lines and towers detected from the image. Pixel-level semantic tags for the target It includes the target's three-dimensional location estimate and its existence confidence level.

[0027] Next, in this embodiment, the system generates high-level control commands based on the state observation vector using formula (2): (2); Formula (2) describes the policy network. (Parameters are) The network's decision-making process is described by the normalized state observation vector. As input, output a normalized, continuous high-level control command. . Control the desired cruise speed along the guide wire direction. and The desired velocity components of the UAV in the horizontal (left-right) and vertical (up-down) directions are controlled separately. The desired yaw rate is controlled to correct the nose direction. The policy network can be a multilayer perceptron or a recurrent neural network with an attention mechanism, which dynamically focuses on key features in the state vector, such as large current pose deviations. Significant forward curvature Or information about obstacles at close range This will improve the rationality of decision-making.

[0028] Furthermore, in this embodiment, the system applies security constraints to the original instructions output by the policy network using formula (3) to ensure their physical feasibility: (3); Formula (3) defines the safety action space constraints. This is the original output of the policy network (Formula 2). and These are the minimum and maximum feasible values ​​for each action dimension, dynamically calculated by the action mask module based on the real-time status of the drone (such as remaining power, structural load, ambient wind speed, and distance to the nearest obstacle). This is a trimming function, its purpose is to trim the original instructions. Each component is limited to the corresponding Within this range, the final, safely executable instructions are obtained. For example, when the drone gets too close to an obstacle on the right, lateral velocity in The upper limit of the component will be set to zero or a negative value to prevent movement to the right.

[0029] Then, in this embodiment, the system uses formula (4) to perform a multi-dimensional performance evaluation of the immediate effects of each state transition (i.e., the execution of an action): (4); Formula (4) defines the total reward signal. , by sparse reward and dense rewards Addition constitutes sparse rewards. It is event-driven. Set a function to indicate task completion (e.g., a value of 1 when a pre-set inspection segment is completed, otherwise 0). This refers to the reward or penalty value corresponding to a key event (e.g., completing an inspection segment rewards +100, a collision incurs a penalty of -500). Dense Rewards Fine-grained feedback is provided in each control cycle, which is a weighted sum of four sub-rewards. The weighting coefficients for each sub-reward. To track rewards, a negative correlation is established with the norm of pose deviation, encouraging precise tracking; To smooth out rewards, the magnitude of changes in continuous control commands is negatively correlated, thus encouraging smooth control. As an efficiency reward, it is positively correlated with cruise speed while meeting tracking accuracy requirements, thus encouraging efficient flight; The system is designed to reward safety by being negatively correlated with the intrusion distance between the drone and the nearest obstacle, and to penalize when the distance is less than the dynamic safe distance, thus encouraging the maintenance of a safe distance.

[0030] Meanwhile, in this embodiment, the system prioritizes the sampling of high-value experiences during experience playback using formula (5): (5); Formula (5) defines the priority experience replay mechanism. This represents the probability of sampling the i-th experience from the experience replay buffer. The importance priority of this experience is determined by the absolute value of its corresponding temporal-difference error (TD-error). Decide. The larger the value, the greater the error in the current network's prediction of the value of the state-action pair, and the higher the potential (value) to learn from this experience. It is an index that controls the intensity of priority, when It degenerates into uniform sampling. It is a parameter used to adjust the priority distribution. It is a very small positive number, used to avoid When the TD-error is zero, the priority is zero and it will never be sampled. This mechanism ensures that experiences with high TD-error (i.e., inaccurate network predictions and rich information) have a higher probability of being selected for training, thereby improving learning efficiency. It should be noted that in formula (5), the absolute value of the temporal difference error is used to define the priority through exponential operation. It is a dimensionless weighting strategy that directs learning towards empirical samples with greater prediction bias and richer potential information.

[0031] Therefore, by combining the above formulas (1) to (5), the system can realize a complete, closed-loop, and adaptive UAV trajectory correction process. Formula (1) provides comprehensive and quantitative environmental and self-state perception; formulas (2) and (3) work together to generate control commands that are both intelligent and flexible and absolutely safe; formula (4) constructs a multi-objective evaluation system that guides the agent to evolve towards "precision, smoothness, efficiency, and safety"; formula (5) optimizes the strategy learned from historical experience and focuses on solving the weaknesses of the current strategy. These mathematical models are interconnected and jointly support the reinforcement learning agent to accumulate experience online and continuously optimize the strategy in complex route corridor environments, ultimately achieving the technical goal of synergistically improving path tracking accuracy, environmental adaptability, and flight safety.

[0032] Step S120: Input the state observation vector into the policy network of the reinforcement learning agent and output high-level control instructions.

[0033] In this context, a reinforcement learning agent is an artificial intelligence program that learns optimal behavioral policies by interacting with its environment and based on reward signals. The policy network is a deep neural network model within the agent used to map current states to optimal actions. High-level control commands refer to abstract control instructions that do not directly manipulate motor speeds or control surface angles, but rather instruct the UAV on desired speed, acceleration, or position increments.

[0034] Specifically, the system can input the normalized state observation vector into an actor network (i.e., policy network) of a deep reinforcement learning agent based on an actor-critic architecture, and output a multi-dimensional, normalized action value through forward pass computation.

[0035] For example, the system's policy network is a fully connected neural network with three hidden layers. For the aforementioned state observation vector, the policy network calculates and outputs a normalized action vector. These correspond to the desired forward speed ratio, lateral speed ratio, vertical speed ratio, and yaw rate ratio, respectively. After inverse normalization, the specific high-altitude control commands are obtained: "Advance at 5 m / s, make minor adjustments to the left at 0.3 m / s, maintain current altitude, and move to the right at 0.3 m / s." Slow yaw.

[0036] Step S130: Send high-level control commands to the low-level flight control system for execution. At the same time, an independent safety monitoring module monitors the flight status in real time and triggers a switch of control when the status exceeds the limit.

[0037] The underlying flight control system refers to the embedded hardware and software system that directly controls the drone's motors, servo mechanisms, and other actuators, responsible for attitude stabilization and low-level control. The independent safety monitoring module is a software unit logically and operationally independent of the reinforcement learning decision loop, specifically designed for flight safety boundary checks and emergency response. Control switching refers to the process by which the system automatically transfers the drone's control authority from the instructions generated by the reinforcement learning agent to a pre-set safety backup control strategy when a security threat is detected.

[0038] Specifically, the system can send high-level control commands to the UAV's open-source flight controller (such as PX4 or ArduPilot) via a communication bus (such as the Controller Area Network CAN bus), which then parses them into specific motor commands. Simultaneously, the safety monitoring module operates in parallel at a higher frequency, checking whether the UAV's real-time position, speed, attitude, and sensor health status exceed preset safety envelopes.

[0039] For example, when a drone violently shakes under strong crosswinds and its roll angle momentarily exceeds the 30-degree safety threshold, the safety monitoring module immediately triggers an alarm. Based on the threat level, the system smoothly switches control from a reinforcement learning strategy to a pre-defined, attitude-recovery-focused PID (Proportional-Integral-Derivative) stabilizing controller within 50 milliseconds (ms), enabling the drone to quickly regain stability and avoid loss of control.

[0040] Step S140: In response to the UAV state transition, collect the new environmental state and calculate the reward according to the reward function, and store the experience data including state, action, reward and next state into the experience replay buffer.

[0041] State transition refers to the process by which a UAV, after executing a high-level control command, moves from the state described by the previous state observation vector to a new environmental state. The reward function is the evaluation criterion function used in reinforcement learning algorithms to evaluate the quality of a single action; its output value is called the reward signal. Experience data is a complete record of a single "state-action-result" interaction process. The experience replay buffer is a data storage area used to store large amounts of historical experience data for offline training of the reinforcement learning agent.

[0042] Specifically, at the end of each control cycle, the system can determine whether the UAV's state has changed significantly due to the previous command. If so, it collects new sensor data to generate a new state observation vector and calls the reward function to calculate a scalar reward value based on multiple dimensions such as tracking accuracy, command smoothness, and distance to obstacles. Subsequently, the state before the decision, the action performed, the reward obtained, and the new state after the decision are packaged into an experience tuple.

[0043] For example, after the drone executes a lateral correction command, its lateral deviation relative to the guide wire decreases from 0.5m to 0.1m. The reward function is calculated as follows: Tracking reward (deviation reduction) +2.0, Smoothness reward (command change is gradual) +0.5, Safety reward (avoiding obstacles) +0.1, Total reward +2.6. The system will record the experience (state) of this interaction. ,action Reward +2.6, Next Status It is stored in an experience replay buffer of a First In First Out (FIFO) queue structure.

[0044] Step S150: Sample historical experience data from the experience replay buffer and update the network parameters of the reinforcement learning agent.

[0045] Network parameters refer to the set of adjustable variables, such as weights and biases, that constitute deep learning models like policy networks and value evaluation networks. Updating these parameters is equivalent to training the model and optimizing its decision-making capabilities.

[0046] Specifically, the system can use a background learning thread, independent of the real-time control thread, to periodically sample a small batch of historical experience data from the experience replay buffer. Then, using reinforcement learning algorithms similar to Twin Delayed Deep Deterministic Policy Gradient (TD3) or Soft Actor-Critic (SAC), the policy gradient and value function gradient are calculated, and the parameters of the policy network and value evaluation network are updated using Stochastic Gradient Descent (SGD) or its variants (such as the Adam optimizer).

[0047] For example, the background learning thread starts every 2000 new experiences, randomly sampling 256 experiences as a batch from a buffer of 50,000 experiences. The algorithm calculates the update direction of the current policy and makes minor adjustments to the network parameters using the Adam optimizer at a learning rate of 0.001, so that the policy can obtain higher rewards when facing similar situations in the future.

[0048] Step S160: During the flight mission, the steps of generating the state observation vector, issuing instructions and monitoring safety, and storing experience are executed cyclically, and the steps of updating network parameters are executed asynchronously with these steps.

[0049] Asynchronous execution means that the network parameter update process and the real-time control loop are processed in parallel without blocking in time. The real-time control loop has extremely high timeliness requirements and must run stably; while the network update has a large computational load, allowing it to be executed in the background during time gaps, and the two do not block each other.

[0050] Specifically, the system can implement a producer-consumer model through multithreaded programming. A high-frequency thread (e.g., 100 Hz) acts as the "producer," strictly and periodically executing steps S110 to S140, namely perception, decision-making, control, and experience storage. Another low-frequency thread acts as the "consumer," awakened when system computing resources are idle (or periodically), executing step S150 to retrieve data from the buffer to train the network. The two threads exchange data through a shared experience replay buffer and parameter server.

[0051] For example, the onboard computer on the drone runs two main threads. The control thread runs stably at a frequency of 100Hz, ensuring that a perception-decision-control loop is completed every 10ms. The learning thread has a lower priority and is scheduled to be executed by the operating system whenever the control thread is idle and the buffer has sufficient data. Each execution may take hundreds of milliseconds to complete one iteration of network parameter updates. The updated network parameters are stored in a shared variable, and the control thread can read this variable in the next cycle to use the latest policy, realizing online hot updates of the policy.

[0052] Therefore, according to the above implementation method, the system achieves its goals through five core steps: state observation, intelligent decision-making, security monitoring, experience learning, and asynchronous optimization. The system comprises several key components: Real-time sensor data from the UAV is collected and compared with a pre-defined reference path model to generate state observation vectors, enabling real-time perception and quantitative representation of the UAV's current pose and environmental characteristics; these state observation vectors are input into the policy network of the reinforcement learning agent, which outputs high-level control commands to achieve intelligent adaptive decision-making based on environmental interaction; high-level control commands are then sent to the underlying flight control system for execution, while an independent safety monitoring module monitors the flight status in real time and triggers a switch of control when the state exceeds limits, ensuring basic flight safety while introducing intelligent decision-making; in response to UAV state transitions, new environmental states are collected, and rewards are calculated based on a reward function; experience data including state, action, reward, and the next state is stored in an experience replay buffer for structured recording and storage of decision-making process data; historical experience data is sampled from the experience replay buffer and the network parameters of the reinforcement learning agent are updated for continuous iterative optimization of the decision model; during flight mission execution, the generation of state observation vectors, command issuance and safety monitoring, and experience storage steps are executed cyclically, with network parameter updates performed asynchronously to achieve parallel processing and non-interference between flight control and model learning.

[0053] Specifically, in the technical solution of this embodiment, to address the problem of "insufficient path tracking accuracy" mentioned in the background technology, a state observation vector is generated and compared with the reference model in real time, providing accurate quantitative input for control decisions and establishing a data foundation for high-precision tracking. To address the problem of "insufficient environmental adaptability and decision flexibility," a reinforcement learning agent outputs high-level control commands and asynchronously updates network parameters based on a reward function and experience replay mechanism, enabling the decision strategy to self-optimize online using historical flight experience, thereby adapting to complex disturbances such as sudden changes in route direction and crosswinds, breaking through the technical limitations of traditional fixed parameter or model-dependent methods with limited adaptive capabilities. To address the problem of "imperfect flight safety assurance mechanism," a safety monitoring module independent of the decision loop is set up, and control is forcibly switched when the state goes out of bounds, constructing a safety redundancy barrier independent of the learning algorithm, effectively avoiding flight risks caused by agent exploration or decision-making errors, and solving the defects of deep coupling between safety mechanisms and decision logic and insufficient reliability in the prior art. Therefore, the technical solution of the present invention solves the technical problem that existing technologies have difficulty in coordinating path tracking accuracy, environmental adaptability and flight safety in complex power line corridor environments, and improves the autonomy, reliability and overall efficiency of UAVs in power line inspection tasks.

[0054] In another embodiment, such as Figure 2The illustration shows a detailed diagram of the adaptive path tracking control process of a drone performing a detailed inspection task within a 220kV high-voltage transmission line corridor. This process, through the closed-loop execution of six core steps (S1 to S6), enables the drone to cope with changing crosswinds, line sag, and dynamic obstacles such as sudden bird attacks in mountainous terrain.

[0055] In step S1, the UAV's onboard system acquires its own pose information and environmental images with centimeter-level precision in real time by fusing data from a real-time dynamic carrier phase differential global navigation satellite system receiver, an inertial measurement unit, and a binocular vision sensor. The system compares and registers this real-time sensor data with a preset reference path model containing the precise three-dimensional coordinates of all towers and conductors along the line segment. Specifically, the system calculates the UAV's current lateral deviation relative to the middle phase conductor as 0.3m (left), vertical deviation as -0.15m (down), and heading angle deviation as -5 degrees, and calculates the rate of change of these deviations over the past 0.1 seconds. Simultaneously, based on visual data, it extracts the curvature of the path 50m ahead as 0.015 per meter (right turn) and detects an insulator target with a confidence level of 0.92 approximately 15m to the right front. All this information is combined into a 42-dimensional state observation vector.

[0056] In step S2, the state observation vector, after normalization, is input into a policy network of a deep reinforcement learning agent based on an actor-critic architecture. The policy network is a three-layer perceptron incorporating an attention mechanism; after processing the input, it outputs a four-dimensional, normalized raw action instruction. These correspond to the desired forward speed, lateral speed, vertical speed, and yaw rate, respectively. Subsequently, the action mask module calculates the feasible range of each action component based on information such as the current ambient wind speed of 8 meters per second (m / s), the remaining battery power of 65%, and the distance to the insulator of 15m, and constrains the original command, ultimately generating a safe high-altitude control command: "Proceed at a speed of 4m / s, correct to the right at a speed of 0.5m / s, descend slowly, and turn slightly to the left to align with the course."

[0057] In step S3, the aforementioned high-level control commands are sent to the UAV's underlying PX4 flight control system via the Controller Area Network (CAN) bus. The flight control system interprets these commands as specific motor speed and control surface commands, driving the UAV to execute them. Simultaneously, a separate safety monitoring module operates in parallel at a frequency of 1000Hz. This module's rule-based verification unit continuously checks whether the UAV's attitude and speed are within their envelope, while the learning-based anomaly detection unit (a lightweight autoencoder) monitors for anomalies in the flight state sequence. In this example, everything is normal, and control remains with the reinforcement learning agent.

[0058] In step S4, after the UAV executes the command, it enters a new state. The system collects new environmental data, generates a new state observation vector, and calls the multi-objective reward function for evaluation. The reward calculation considers the reduced pose deviation (tracking reward +1.2), smooth control changes (smoothing reward +0.3), and the maintained safe distance (safety reward +0.1). However, because the cruise speed is low at this time, the efficiency reward is 0, so the total immediate reward obtained for this action is +1.6. Subsequently, the system packages the state observation vector before the action, the executed high-level control command, the obtained reward +1.6, and the new state observation vector into a single empirical data point and stores it in a shared circular empirical buffer.

[0059] In step S5, an independent low-frequency learning thread is automatically triggered after detecting that the circular experience buffer has been filled with 5000 experiences. This thread employs a priority experience replay mechanism, sampling 256 historical experiences from the buffer (192 from the high-priority region and 64 from the ordinary region), and applying importance sampling weights to correct for biases. Based on this sampled data, the thread performs offline policy optimization, calculating gradients using a dual-delay deep deterministic policy gradient algorithm to update the parameters of the policy network and the value evaluation network. After the update is complete, the thread uses a soft update strategy (mixed coefficients) to update the parameters. This will slowly synchronize the parameters of the main network to the corresponding target network.

[0060] In step S6, steps S1 to S4 are executed cyclically at a frequency of 100Hz in the real-time control thread, forming a stable and highly responsive flight control loop. Step S5, on the other hand, is executed asynchronously by an independent low-priority learning thread. The two threads exchange data and model parameters through a shared experience buffer and parameter server, without blocking each other. In this way, while continuously performing inspection flights, the UAV can dynamically optimize its decision model by constantly utilizing the newly accumulated flight experience. As a result, when faced with challenges such as changes in line curvature and increased wind speed disturbances that were not accurately modeled beforehand, it exhibits increasingly accurate, smooth, and adaptive path tracking capabilities.

[0061] In some embodiments, real-time sensor data from the UAV is collected and compared with a preset reference path model to generate a state observation vector, including: By fusing satellite positioning signals with inertial measurement unit data, the attitude information of the UAV is obtained.

[0062] Among them, pose information is a six-degree-of-freedom state quantity that describes the position and attitude of the UAV in three-dimensional space, which usually includes three-dimensional coordinates (such as longitude, latitude, and altitude) and three-dimensional attitude angles (roll angle, pitch angle, and yaw angle).

[0063] Specifically, the system can use a loosely coupled or tightly coupled filtering algorithm (such as an extended Kalman filter) to fuse high-precision position and velocity information from an RTK (Real-Time Dynamic Carrier Phase Differential) GNSS (Global Navigation Satellite System) receiver with high-frequency angular velocity and acceleration information from an IMU (Inertial Measurement Unit) to estimate and output the UAV's real-time attitude at a frequency of 100Hz or higher. For example, in an inspection operation, the system calculates the following real-time attitude information: longitude 118.123456°, latitude 24.567890°, altitude 350.5m, roll angle -0.5°, pitch angle 1.2°, and yaw angle 45.3°.

[0064] Environmental image data is acquired through a visual sensor.

[0065] Visual sensors refer to devices used to acquire optical images of the environment, such as monocular cameras, binocular stereo cameras, or infrared thermal imagers. Environmental image data consists of digital images captured by these sensors that contain information about the lines and the surrounding scene.

[0066] Specifically, the system can control a binocular stereo camera mounted on the drone's gimbal to simultaneously acquire high-definition digital images (e.g., 1920 pixels multiplied by 1080 pixels resolution) for the left and right eyes at a frame rate of 30 frames per second (fps), and transmit the image data to the onboard computing unit via a high-speed serial bus (e.g., MIPI CSI-2). For example, while the drone is in flight, the binocular camera captures a stereo image pair containing three power transmission lines in front, a power tower, and a mountain in the background. The image format is RGB (red, green, blue) three-channel, with a bit depth of 8 bits per channel.

[0067] Based on the pose information, the model is registered with a reference path model that includes the spatial topology of the route, and the pose deviation of the UAV relative to the reference path and the rate of change of the pose deviation are calculated.

[0068] Registration refers to the process of aligning the real-time measured UAV pose to the global coordinate system defined by the reference path model. Pose deviation is the difference between the UAV's current position and attitude and the ideal position and attitude of the nearest point on the reference path. The rate of change is how quickly this deviation changes over time.

[0069] Specifically, the system can find the nearest reference waypoint to the UAV's current position by querying a reference path model (usually composed of a series of ordered 3D waypoints), and calculate the projection of the vector from that waypoint to the UAV in the lateral, normal, and heading coordinate systems of the path, thereby obtaining the lateral deviation, vertical deviation, and heading angle deviation. The rate of change can be calculated by the difference between the current deviation and the deviation of the previous cycle. For example, after registration calculation, the system finds that the UAV's current lateral deviation is +0.25m (indicating it is on the right side of the reference path), the vertical deviation is -0.10m (indicating it is below the reference altitude), and the heading angle deviation is +2.1° (indicating the nose is turned to the right). At the same time, the system calculates that the rate of change of the lateral deviation in the past 0.1 seconds is +0.05 meters per second (m / s), indicating that the UAV is deviating to the right.

[0070] Based on environmental image data, geometric look-ahead features of the path are extracted. These geometric look-ahead features are used to characterize the curvature properties of the path and the changing trend of these curvature properties.

[0071] Among them, geometric look-ahead features are quantitative parameters derived from images to describe the geometric shape of a path ahead, and are designed to provide predictive information for drones.

[0072] Specifically, the system generates a 3D point cloud of the foreground scene by performing stereo matching on binocular images. It then segments point cloud clusters belonging to the guide lines and fits these clusters using polynomial or spline curves. By analyzing the mathematical expression of the fitted curve, the system calculates the curvature value of the curve at a predicted point a distance (e.g., 20m) in front of the drone and calculates the gradient of this curvature along the path. For example, the system extracts a predicted curvature of 0.02 m / s (1 / m) at a point 20m away from the drone, indicating a slight right turn; simultaneously, it calculates a curvature gradient of 0.001 m / s (1 / m²), indicating a slowly increasing turning tendency.

[0073] Environmental image data is input into a pre-trained deep neural network to extract forward-looking features of environmental threats, including the type, location, and presence confidence information of targets within the route corridor.

[0074] Among them, pre-trained deep neural networks refer to convolutional neural network models that have been trained on large labeled datasets and have the ability to detect objects and segment semantics, such as YOLO (You Only Look Once, a single-stage object detection algorithm) or DeepLab series models.

[0075] Specifically, the system can input real-time acquired single-frame or multi-frame environmental images into a lightweight neural network model deployed on an airborne computing unit. This model outputs the semantic label, two-dimensional pixel coordinates, and confidence score for each pixel or bounding box in the image, along with the corresponding category. Combining the camera's intrinsic parameters and the UAV's current pose, the two-dimensional detection results can be converted into a three-dimensional position estimate relative to the UAV. For example, the neural network detects an "insulator" target in the image, with the center pixel coordinates of the two-dimensional bounding box at (850, 300) and a confidence score of 0.95. Incorporating depth information, it estimates that the insulator is located approximately 15m to the right front and approximately 2m below the UAV. Simultaneously, a "bird" target is detected in the distance with a confidence score of 0.70.

[0076] The pose deviation, the rate of change of pose deviation, geometric look-ahead features, and environmental threat look-ahead features are combined to generate a state observation vector.

[0077] Combination refers to concatenating multiple feature values ​​with different dimensions and physical meanings into a one-dimensional numerical vector in a predetermined order.

[0078] Specifically, the system can perform data concatenation operations to sequentially concatenate the calculated lateral deviation, vertical deviation, and heading angle deviation scalars, their corresponding three rate-of-change scalars, an extracted curvature value and a curvature change gradient scalar, and all detected threat target information (each target is represented by four scalars: category code, relative distance, relative azimuth, and confidence score). If the number of targets is not fixed, the top N targets with the highest confidence scores can be selected, or fixed-dimensional features can be generated through attention pooling. For example, the system combines all the above features into a single state observation vector: (Other rates of change) (Insulator: Category Code 1, (Birds: Category code 5, The vector contains a total of 40 dimensions.

[0079] Therefore, according to the above implementation method, the system can comprehensively utilize precise satellite / inertial navigation, stereo vision geometric perception, and deep learning-based environmental semantic understanding to construct a high-dimensional state representation that is comprehensive in information and includes real-time state and forward prediction, providing a solid and rich perception foundation for subsequent reinforcement learning agents to make safe, accurate, and predictive flight decisions.

[0080] In some embodiments, the state observation vector is input into the policy network of the reinforcement learning agent, and high-level control instructions are output, including: The state observation vector is input into the policy network of the reinforcement learning agent. The policy network is a multilayer perceptron or recurrent neural network that includes an attention mechanism.

[0081] Among them, the attention mechanism is a neural network component that simulates the way human attention is allocated. It can dynamically calculate the importance weights of features in different dimensions of the input vector, enabling the network to focus on key information (such as features related to the nearest obstacle) when dealing with complex states. A multilayer perceptron is a feedforward neural network composed of stacked fully connected layers and activation functions. A recurrent neural network is a neural network with internal states, suitable for processing sequential data.

[0082] Specifically, the system can input the normalized state observation vector into a multilayer perceptron with three hidden layers. After the second hidden layer, the network introduces a multi-head self-attention module to calculate the contribution of different feature dimensions in the state vector to the current decision and to perform weighted fusion of the features, thereby making the decision focus more on imminent threats (such as approaching birds) and critical tracking errors (such as increased lateral bias). For example, a usable policy network structure is: Input layer (40-dimensional) - Fully connected layer (256 neurons) - ReLU (Modified Linear Unit) activation function - Fully connected layer (128 neurons) - Multi-head self-attention layer (4 heads) - Fully connected layer (64 neurons) - ReLU activation function - Output layer (4-dimensional).

[0083] The input state observation vector is processed by the policy network to output normalized raw action commands, which include the desired forward velocity component, lateral velocity component, vertical velocity component and yaw rate component.

[0084] Normalization is a data processing method that linearly scales data to a fixed range (e.g., ...). This facilitates stable training and output interpretation of the neural network. The raw action instructions are the initial output of the policy network without any safety or physical constraints.

[0085] Specifically, the system can use the forward propagation computation of the policy network to apply the Tanh (hyperbolic tangent) activation function in the output layer, thereby restricting the values ​​of the four output neurons to a certain value. Within the interval, these correspond to the normalized command values ​​of the aforementioned four velocity components. For example, for a specific state observation vector, after the policy network performs forward computation, it outputs a four-dimensional vector. This means that, under normalized scaling, the network suggests the following original actions: forward velocity component +0.8, lateral velocity component -0.1 (leftward), vertical velocity component +0.05 (ascent), and yaw rate component +0.02 (right turn).

[0086] The motion mask module dynamically calculates the feasible range of each motion component based on the real-time status information of the UAV, including remaining power, structural load, ambient wind speed, and distance to obstacles.

[0087] The motion mask module is an independent functional module that calculates and generates the permissible range of motion in real time based on the current physical state of the UAV and environmental constraints. Structural load refers to the structural stress borne by the UAV fuselage under the influence of factors such as airflow, and can usually be estimated from inertial measurement unit data. The feasible range defines the lower and upper limits of the values ​​allowed for each motion component at the current moment.

[0088] Specifically, the system can map real-time status information into restrictions on each action component based on a set of predefined safety rules. For example, when the remaining battery power is below 20%, the maximum forward speed is limited; when structural load (such as vibration acceleration) exceeds a threshold, acceleration commands in all directions are limited; when the ambient wind speed is too high, the feasible range of lateral movement is reduced according to the wind resistance model; when the distance to the nearest obstacle is less than the safe distance, the upper limit of the action component in the direction of the obstacle is set to zero or a negative value (i.e., prohibiting approach or requiring distance). For example, suppose the current distance between the drone and an insulator on the right is 8m, and the preset dynamic safe distance is 10m. The action masking module calculates the upper limit of the feasible range of the current lateral speed component as 0 based on the rule "prohibit movement towards the obstacle" (because moving to the right would bring the drone closer to the obstacle). At the same time, with the remaining battery power at 30%, the module limits the maximum forward speed component to 0.9. Therefore, the feasible range of the current four action components may be calculated as: forward speed lateral speed Vertical velocity yaw rate .

[0089] Based on the feasible range calculated by the action mask module, the original action instructions output by the policy network are constrained to generate higher-level control instructions.

[0090] Among them, constraint processing refers to the operation of restricting the value of the original action instruction to within the feasible range specified by the action mask module.

[0091] Specifically, the system can perform constraint processing through component-by-component comparison and truncation operations. For each raw action command component output by the policy network, it checks whether it falls within the feasible range [lower limit, upper limit] of the corresponding component given by the action mask module. If it exceeds this range, the command value is truncated (i.e., set to) the nearest boundary value; if it falls within the range, the original value is retained. The processed result is the final high-level control command issued. For example, the raw action command output by the policy network is... After considering the above feasible scope A component-by-component comparison revealed that all original instruction values ​​were within acceptable limits. Therefore, no truncation was necessary for the original instructions. This is used as the final high-level control command. If the original lateral speed command is +0.2 (i.e., to the right), it exceeds... If the upper limit is 0, it will be truncated to 0, and the lateral velocity component in the generated high-level control command will become 0.

[0092] Therefore, according to the above implementation method, the system can make flexible and adaptive decisions using reinforcement learning agents, and at the same time, it can seamlessly and forcibly inject the real-time physical state and safety constraints of the UAV into the decision-making process through a parallel, rule-based action masking module. This ensures that every high-level control command generated complies with the current safety and physical feasibility boundaries before it is issued, fundamentally avoiding the possibility that the agent may output dangerous or infeasible commands due to exploration or misjudgment, and significantly improving the overall security of the system.

[0093] In some embodiments, higher-level control commands are sent to the lower-level flight control system for execution, while an independent safety monitoring module monitors the flight status in real time and triggers a switch of control when the status exceeds the limits, including: A pre-configured security monitoring module with a hybrid architecture integrates a rule-based runtime verification system and a learning-based anomaly detector.

[0094] The hybrid architecture refers to the fact that the security monitoring module does not use a single technical principle, but integrates deterministic rule reasoning and data-driven machine learning methods to balance reliability, real-time performance and the ability to detect unknown anomalies.

[0095] Specifically, during the software initialization phase, the system can load and instantiate two parallel monitoring units: one is a rule engine containing a series of if-then logical judgment statements, and the other is an inference engine that loads a pre-trained machine learning model file. The two units share the same real-time flight status data input but perform independent judgments. For example, at startup, the system reads the list of geofence vertex coordinates, attitude angle, and velocity threshold parameters from the configuration file to initialize the rule verification system; simultaneously, it loads a lightweight autoencoder neural network model that has been trained on a large amount of normal flight data from a model file to initialize the anomaly detector.

[0096] A rule-based runtime verification system performs continuous state verification on predefined multi-level security attributes, including airspace geofencing, attitude angle envelope, velocity envelope, and sensor data validity.

[0097] In this context, an airspace geofence is a predefined virtual boundary in three-dimensional space outside which drones are prohibited from flying, typically represented by a polygon or cube. The attitude angle envelope is the range of permissible roll, pitch, and yaw angles. The velocity envelope is the range of permissible forward, lateral, and vertical velocities. Sensor data validity refers to checking the signal status and data update rate of key sensors (such as Global Navigation Satellite Systems and Inertial Measurement Units) to ensure they are normal.

[0098] Specifically, during runtime verification, the system receives the latest UAV status information within each control cycle (e.g., 10ms) and sequentially performs the following checks: calculates whether the current position is inside all geofence polygons; determines whether the absolute values ​​of the current roll, pitch, and yaw angles are less than their respective set thresholds (e.g., 25°, 25°, 30°); determines whether the synthesized velocity is less than the maximum permissible velocity (e.g., 15m / s); and checks whether the Global Navigation Satellite System (GNSS) positioning status flag is a fixed solution and whether the inertial measurement unit (INS) data is within a physically reasonable range (e.g., acceleration does not exceed 20 times the gravitational acceleration g). For example, in one verification, if the system detects that the UAV's current longitude and latitude exceed the mission-defined rectangular geofence (118.12~118.13°E, 24.56~24.57°N), it immediately triggers a "boundary violation" alarm and records the infringing latitude and longitude coordinates. .

[0099] A learning-based anomaly detector is loaded with an autoencoder or a single-class support vector machine model to construct a normal behavior benchmark based on the historical normal flight state sequence of the UAV, and anomaly pattern detection is performed on the real-time flight state sequence.

[0100] Among them, an autoencoder is an unsupervised learning model that attempts to compress and reconstruct input data using a neural network, and detects anomalies by comparing the differences between the reconstructed data and the original data. A single-class support vector machine is a machine learning model trained only on normal samples to determine whether a new sample belongs to a "normal" distribution. Normal behavior benchmarks are the typical state sequence patterns of drones under fault-free and interference-free conditions, characterized by the internal parameters of these models.

[0101] Specifically, the anomaly detector receives a short sequence of states over a short period (e.g., state observation vectors at 100 points within the last second). An autoencoder model attempts to reconstruct this input sequence and calculates the reconstruction error at each point in time. If the average reconstruction error exceeds a threshold calculated based on historical normal data, the current sequence is considered anomaly. A single-class support vector machine model directly calculates the position of the current state sequence in the high-dimensional feature space defined by the model, determining whether it falls within the "normal" region. For example, the system uses an autoencoder with an encoder-decoder structure. Inputting a 40-dimensional state vector sequence of the last second (100 points), the autoencoder outputs a reconstructed sequence. The calculated average reconstruction error for this sequence is 0.85, while according to offline statistics, the 99th percentile of this error during normal flight is 0.50. Since 0.85 > 0.50, the anomaly detector triggers a "pattern anomaly" alarm, indicating a possible interference that the model has not been trained on (such as sudden strong turbulence or a latent sensor malfunction).

[0102] In response to an alarm triggered by a rule-based runtime verification system or a learning-based anomaly detector, a control switch operation is performed based on the threat level of the alarm. The control switch operation includes a smooth transition from the policy network of the reinforcement learning agent to a preset backup controller, or a switch to a preset emergency return trajectory tracking controller.

[0103] Threat level is a pre-defined classification based on alarm type and severity, used to determine the intensity of emergency response. Smooth transition refers to a gradual change in control commands during a switchover, not an instantaneous replacement, but rather a continuous change in commands over a very short time (e.g., within 50ms) through interpolation, avoiding impact on the drone. The preset backup controller is typically a simple and reliable traditional controller, such as a proportional-integral-derivative controller. The emergency return-to-home trajectory tracking controller is specifically designed to guide the drone back to its takeoff point along a pre-defined safe path.

[0104] Specifically, the system can predefine a threat level mapping table. For example, "sensor failure" and "severe attitude overrun" can be defined as Level 1 threats (highest), immediately switching to the emergency return-to-home controller. "Minor boundary violation" and "pattern detection anomaly" can be defined as Level 2 threats. Initially, a smooth transition from the reinforcement learning policy network to a conservative proportional-integral-derivative (PI-DI) backup controller focused on maintaining the current position and attitude is attempted. If normal operation is not restored within a specified time (e.g., 2 seconds), the threat level escalates to Level 1, triggering a return-to-home mechanism. For instance, when a rule-based system triggers an "airspace violation" alert (Level 2 threat), the system begins a smooth transition of control in the next control cycle. Within 50ms, the source of higher-level control commands linearly transitions from the output of the reinforcement learning policy network to the output of the backup PI-DI controller, which attempts to stabilize the UAV at its current altitude and hover. If the UAV still has not returned to the geofence after 5 seconds, the system generates a straight return trajectory pointing to the takeoff point and switches to the trajectory tracking controller for automatic return-to-home.

[0105] Therefore, according to the above implementation method, the system can construct a defense-in-depth security monitoring system. It ensures the absolute avoidance of known risks through rule-based rapid hard boundary checks, captures unknown risks that are difficult to describe using rules through learning-based anomaly detection, and executes tiered and smooth control takeover strategies based on the severity of the risks. This fully leverages the autonomous decision-making advantages of reinforcement learning agents while providing them with a robust "safety cage," ensuring that any single point of failure or decision-making error will not jeopardize flight safety, thus improving the robustness and reliability of the system in complex real-world environments.

[0106] In some embodiments, the steps of acquiring new environmental states and calculating rewards based on reward functions, and storing experience data including states, actions, rewards, and the next state into an experience replay buffer include: Based on the new environmental sensing data collected after the UAV completes the state transition, a state perception and feature extraction process is executed to generate a new state observation vector.

[0107] Among them, the state perception and feature extraction process refers to the complete data processing chain that calculates standardized state information for decision-making from new raw sensor data.

[0108] Specifically, the system can trigger a new round of sensor data acquisition and processing after the UAV has executed a high-level control command and stabilized (e.g., 50ms after the command is issued). This process reuses the fusion, registration, and feature extraction methods defined in step S110 to calculate the UAV's pose deviation, deviation change rate, geometric look-ahead features, and environmental threat look-ahead features at the new moment, and combines these features into a new state observation vector. For example, after the UAV executes a command to "move right 0.2m", the system acquires new data, calculates a new lateral deviation of 0.05m, a vertical deviation of -0.08m, and a heading angle deviation of 1.8 degrees, and generates the corresponding new state observation vector. .

[0109] A multi-objective reward function is invoked to evaluate the state transition process and calculate the immediate reward signal. The multi-objective reward function combines event-driven sparse rewards with periodic dense rewards.

[0110] Among them, a multi-objective reward function refers to a function that integrates multiple optimization objectives (such as tracking accuracy, flight smoothness, efficiency, and safety) into a single scalar reward value. Event-driven sparse rewards refer to rewards provided for specific, infrequent events (such as completing a task or a collision). Periodically provided dense rewards refer to fine-grained rewards provided in each control cycle based on the current state.

[0111] Specifically, at the end of each control cycle, the system first determines whether a sparse reward event has occurred. Regardless of whether a sparse reward event is triggered, the dense reward is calculated. Total reward signal. From sparse rewards and dense rewards Adding them together, we get: For example, if the drone neither completes its phased task nor causes a collision within a control cycle, then sparse rewards may occur. The system calculates the dense reward for the current period. Therefore, the total reward signal given to the agent in this cycle is +1.5.

[0112] The triggering conditions and reward types for sparse rewards include: positive rewards are generated in response to inspection task progress events that complete a preset distance; negative rewards are generated in response to collision events or task timeout events.

[0113] Among them, the preset distance inspection task progress event refers to the cumulative mileage of the drone flying along the route reaching a preset milestone (e.g., every 100m of flight). A collision event refers to the drone making physical contact with any obstacle (such as power lines, poles, or trees) or being detected by the system as having a very high probability of collision risk. A task timeout event refers to the actual flight time exceeding the maximum allowable time allocated for this inspection task.

[0114] Specifically, the system maintains a flight mileage accumulator and a task timer. When the mileage accumulator exceeds a preset milestone (e.g., 100m), a large positive reward value (e.g., +1000) is immediately awarded, and the accumulator is reset to zero. When the collision sensor is triggered or the vision system detects an unavoidable collision, a very large negative reward value (e.g., -10000) is awarded, and the training round is terminated. When the task timer expires, a large negative reward value (e.g., -500) is awarded. For example, if a drone accumulates 305m of flight during an inspection, the system awards a sparse reward of +1000 when it crosses the 300m milestone. Subsequently, during an attempt to traverse dense tree branches, a collision occurs, triggering the collision sensor. The system immediately awards a sparse reward of -10000 and marks the training round as over.

[0115] Dense rewards are calculated in each control cycle and include: a tracking reward for incentivizing reduced pose deviation; a smoothing reward for incentivizing smooth changes in control commands; an efficiency reward for incentivizing increased cruise speed while meeting tracking accuracy requirements; and a safety reward for incentivizing maintaining a safe distance from obstacles. When the distance between the UAV and an obstacle is less than the dynamic safe distance, the safety reward is penalized.

[0116] The dynamic safety distance is the minimum allowable distance that is dynamically adjusted based on the drone's current speed and the type of obstacle. The faster the speed, the larger the required safety distance is usually.

[0117] Specifically, dense rewards It is the sum of the four reward items: .

[0118] Tracking reward items It is inversely proportional to the square of the overall pose deviation. For example, (lateral deviation^2 + vertical deviation^2 + heading deviation factor), where k1 is a positive weighting coefficient. The smaller the deviation, the closer this reward is to 0 (the smaller the penalty).

[0119] Smooth reward items It is inversely proportional to the magnitude of the change in continuous control commands. For example, (∆Forward speed^2 + ∆Lateral speed^2 + ∆Vertical speed^2 + ∆Yaw speed^2), where k2 is a weighting coefficient. The more drastic the change in command, the greater this penalty.

[0120] Efficiency Rewards When the overall pose deviation is less than a certain accuracy threshold (e.g., 0.3m), this term is proportional to the forward velocity component; otherwise, it is 0. For example, Forward speed (when deviation < threshold), where k3 is the weighting coefficient.

[0121] Safety Rewards The intrusion distance is proportional to the intrusion distance of the drone and the nearest obstacle. Intrusion distance = Dynamic safe distance - Actual distance. When the actual distance is not less than the dynamic safe distance, the intrusion distance is 0. When the actual distance is less than the dynamic safety distance, the intrusion distance is positive. The intrusion distance, where k4 is a larger weighting coefficient that forms the penalty. For example, assuming the current lateral deviation is 0.1m, vertical deviation is 0.05m, forward speed is 0.8 (normalized), the distance to the nearest obstacle (insulator) is 7m, and the current dynamic safety distance is 10m, then the intrusion distance is 3m. If the command change is very small, the calculation can be obtained as follows: Dense reward sum This negative reward primarily stems from safety penalties, which will drive the agent to avoid obstacles in the future.

[0122] The state observation vector before migration, the executed high-level control instructions, the calculated reward signal, and the new state observation vector are encapsulated into empirical data and written into the empirical replay buffer.

[0123] Encapsulation refers to packaging these four elements into a data structure with a fixed format. Writing refers to adding this data structure to the storage space of the experience playback buffer.

[0124] Specifically, the system stores these four elements as a tuple (S_t, A_t, R_t, S_t1) in memory, where S_t is the state before the transition, A_t is the action performed, R_t is the reward obtained, and S_t1 is the new state. The system then appends this tuple to the end of the experience replay buffer (usually implemented as a fixed-length first-in-first-out queue). If the buffer is full, the oldest experience is overwritten. For example, the system generates an experience where S_t is a 40-dimensional vector, A_t is... R_t is -1.35, and S_t1 is a new 40-dimensional vector. The experience is converted into a NumPy array or a PyTorch tensor and then pushed into a circular buffer with a maximum capacity of 100,000 experiences.

[0125] Therefore, according to the above implementation method, the system can transform complex flight performance requirements (precision, smoothness, efficiency, and safety) into scalar feedback signals that the reinforcement learning agent can understand and optimize through a designed multi-objective reward mechanism. Sparse rewards set the success or failure criteria for key tasks, while dense rewards provide fine-grained guidance for each step. Combined with an experience replay buffer, the system can effectively store the massive amounts of interaction data generated during online flight, transforming them into training samples that the agent can repeatedly learn from and summarize patterns, thereby driving the policy network to continuously evolve towards a better, safer, and more efficient direction.

[0126] In some embodiments, sampling historical experience data from an experience replay buffer to update the network parameters of the reinforcement learning agent includes: Maintain a priority experience replay buffer, which is logically divided into a high-priority storage area and a normal storage area.

[0127] The priority experience replay buffer is an improved experience storage data structure. It not only stores experiences but also assigns a priority score to each experience to guide the sampling process, ensuring that experiences more valuable for learning are selected for training with a higher probability. Logical partitioning refers to dividing the same physical storage space into two logical parts, rather than two independent physical storage units, through different indexes or labels in software implementation.

[0128] Specifically, the system can initialize a fixed-size first-in-first-out (FIFO) queue in memory as an experience replay buffer. Simultaneously, it maintains two independent index lists or bitmaps, pointing to the positions of currently stored experience data in the buffer, marked as "high priority" and "normal," respectively. For example, the system initializes a circular buffer with a maximum capacity of 100,000 experiences. It also creates two empty lists, `high_priority_indices` and `normal_indices`, to dynamically record the position indices of the two types of experiences in the buffer.

[0129] Based on the time-series differential error corresponding to each piece of stored experience data, the priority score of the experience data is dynamically calculated and maintained, and the experience data is stored in the selected storage area according to the priority score.

[0130] Temporal difference error is a scalar value used in reinforcement learning algorithms to measure the difference between the current value estimate and a better estimate of a state-action pair. It reflects the "surprise" or value of the experience in updating network parameters; a larger absolute value of the error usually means more can be learned from the experience. Priority score is a positive number calculated based on temporal difference error and is used to quantify the importance of the experience.

[0131] Specifically, when a new experience (S_t, A_t, R_t, S_t1) is stored in the buffer, the system performs a forward propagation on this experience using the current value evaluation network and calculates the absolute value of its temporal difference error. The initial priority score p of this experience can be calculated as follows: ,in It is a very small positive constant (e.g., 0.01) used to ensure that even if the empirical error is 0, there is a very small probability that it will be sampled. The system sets a priority threshold. .if If the time-series difference error absolute value is 0.85, then the experience is stored in the high-priority storage area (i.e., its index is added to the high_priority_indices list); otherwise, it is stored in the normal storage area (the index is added to the normal_indices list). For example, if the absolute value of the time-series difference error of a new experience is 0.85, then... Then the initial priority score p = 0.86. If the system sets a threshold... Since 0.86 > 0.5, this experience is marked as high priority, and its index (such as buffer position 55001) is recorded in the high priority index list.

[0132] Based on the priority scores of the empirical data, a priority-based random sampling strategy is adopted to sample historical empirical data from high-priority storage areas with a higher probability than sampling from ordinary storage areas.

[0133] Among them, the priority-based random sampling strategy is a non-uniform sampling method in which the probability of each experience being sampled is proportional to its priority score, rather than being completely random.

[0134] Specifically, when the system needs to perform a training iteration, it first determines the number of experience records to sample from the high-priority storage area. And the number of experience records sampled from the general storage area ,and For example, setting the batch size to 256, where... , Next, sampling probabilities are calculated for each experience in both the high-priority index list and the regular index list, based on its priority score (which may be renormalized within the high-priority region). A corresponding number of experience indices are then extracted using a roulette wheel algorithm or random sampling. Finally, the complete experience data corresponding to these indices is retrieved from the main buffer, forming a training batch. For example, if there are currently 20,000 experiences in the high-priority region, and the total priority score is... To extract 192 experiences from this area, the system calculates the sampling probability of each experience as follows: Then, weighted random sampling is performed to extract 192 indices. Similarly, 64 indices are drawn evenly or fractionally from the general area (assuming 80,000 experiences). The final batch consists of 256 experiences.

[0135] Importance sampling weights are applied to data sampled from high-priority storage areas.

[0136] The importance sampling weight is a multiplicative coefficient used to correct for data distribution bias caused by non-uniform sampling (i.e., high-priority experience is oversampled), ensuring the unbiasedness of gradient estimation.

[0137] Specifically, for each piece of experience sampled from the high-priority region, the system calculates its importance sampling weight. A common calculation method is Where N is the total number of empirical values ​​in the buffer. It is the probability that the experience is sampled (related to its priority score). It is a hyperparameter between 0 and 1, used to adjust the degree of correction. This is the maximum value of all weights in the current batch, used for normalization. This weight... This will be multiplied by the loss function or gradient calculated from that experience. For example, suppose the total number of experiences in the current buffer is N = 100,000, and the hyperparameter... For a sampling probability in the high priority region Based on experience, its unnormalized weights are calculated as follows: Assuming the maximum weight calculated in this batch is 0.25, then the final importance sampling weight for this piece of experience is... When calculating the gradient of this experience later, the gradient value will be multiplied by 0.764.

[0138] Offline policy optimization is performed based on the sampled historical experience data. The policy gradient is calculated to update the policy network parameters of the reinforcement learning agent, the value function gradient is calculated to update the value evaluation network parameters of the reinforcement learning agent, and the target network parameters corresponding to the policy network and the value evaluation network are updated synchronously based on the soft update policy.

[0139] Offline policy optimization refers to optimizing the policy using historically collected empirical data (rather than data generated by real-time interactions with the current policy). The policy gradient is the gradient of the objective function (expected cumulative reward) with respect to the policy network parameters, indicating how to adjust the parameters to increase future rewards. The value function gradient is the gradient of the estimated value function (evaluating the quality of a state or state-action pair) with respect to the value evaluation network parameters. The target network is a copy of both the policy network and the value evaluation network; its parameters are updated more slowly to provide a stable learning objective and reduce training oscillations. Soft updates mean that during each training iteration, the parameters of the target network are scaled up proportionally (e.g., 0.995) towards the current network parameters, rather than being directly copied.

[0140] Specifically, the system can employ the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm. Using the sampled batch data, it first calculates the current value assessment network for each... Q-value estimation, and the pairing of the two target networks Q value (where Based on the target policy network (Generate and add pruned noise). Then calculate the temporal difference objective value, and subsequently calculate the loss function (e.g., mean squared error loss) and gradient of the value evaluation network. Update the value evaluation network parameters using stochastic gradient descent (e.g., Adam optimizer). Next, calculate the loss function (i.e., negative Q-value expectation) of the policy network, calculate the policy gradient, and update the policy network parameters. Finally, perform a soft update: target policy network parameters... The target value network parameters are also updated similarly, among which... It is a very small number, such as 0.005. For example, using the Adam optimizer with a learning rate of 0.001, the parameters of the value network and the policy network are updated. Then, the soft update coefficients are set. After the update, each parameter of the target policy network becomes 0.995 * original target parameter + 0.005 * new policy network parameter.

[0141] Therefore, according to the above implementation method, the system can intelligently select the most informative historical experiences (i.e., experiences with large sequential difference errors) for current policy optimization through a priority experience replay mechanism, focusing on learning scenarios that have not yet been mastered or are prone to errors, thereby greatly improving data utilization efficiency and training speed. Through importance sampling weight correction, the statistical unbiasedness of gradient estimation is ensured while favoring important samples, maintaining the stability of the learning process. Combining offline policy optimization and soft update strategies, the system can safely and stably learn from historical data, continuously iteratively optimizing decision-making strategies and value assessments, ultimately enabling the reinforcement learning agent to exhibit more accurate and robust control performance in complex line inspection tasks.

[0142] In some embodiments, the steps of generating state observation vectors, issuing instructions and conducting security monitoring and experience storage are executed cyclically, and the steps of updating network parameters are executed asynchronously with these steps, including: Deploy a producer-consumer concurrent model, with the producer thread serving as a high-frequency real-time control thread and the consumer thread serving as a low-frequency offline learning thread.

[0143] The producer-consumer concurrency model is a classic software design pattern used to solve synchronization and efficiency problems when one thread (producer) generates data and another thread (consumer) processes it. In this model, the two threads share a data buffer; the producer writes data to the buffer, and the consumer reads data from it. They can run independently and asynchronously.

[0144] Specifically, the system can create two independent execution threads using multithreading libraries in modern programming languages ​​such as C++ or Python. One thread is given a high real-time priority and bound to a specific CPU core to ensure stable cyclical operation; this thread acts as the producer. The other thread is set to a lower priority and acts as the consumer, scheduled for execution when system computing resources are idle. For example, on an onboard computer based on a Robot Operating System (ROS), the system creates a real-time thread named `control_loop` (the producer) and a background thread named `learning_thread` (the consumer). The `control_loop` thread has its priority set to `SCHED_FIFO` (First-In-First-Out scheduling) to ensure timing accuracy, while `learning_thread` uses the default `SCHED_OTHER` (Other scheduling) policy.

[0145] The producer thread performs the following actions in a fixed control cycle: generating state observation vectors, performing forward inference based on the current policy network parameters to output higher-level control commands, verifying the flight status of the independent safety monitoring module, and writing the generated experience data into a shared circular experience buffer, thus forming a control loop.

[0146] The fixed control cycle refers to the strict time interval between the start of two loops of the producer thread, which is a key parameter to ensure the stability and real-time performance of the control system. The shared circular experience buffer is a first-in, first-out memory area with circular overlay characteristics, shared between the producer and consumer threads, used to temporarily store the experience data generated by the producer thread.

[0147] Specifically, after the producer thread starts, it enters an infinite loop driven by a high-precision timer. At the beginning of each cycle, the thread first collects sensor data and executes step S110 to generate the current state observation vector. Next, the thread reads the latest policy network parameters from shared memory, loads them into the policy network model for forward inference, and outputs higher-level control instructions. Then, the thread sends instructions to the flight controller and calls the independent safety monitoring module to verify the current flight status. Finally, before the cycle ends, the thread processes the experience data generated from this interaction. Write to the circular buffer. If the buffer is full, overwrite the oldest data. For example, set the control period to 10ms. The producer thread is woken up every 10ms and completes all sensing, decision-making, monitoring, and buffer write operations within 2-3ms. A circular buffer with a capacity of 5000 experience points can cache data generated within approximately 50 seconds, providing the consumer thread with sufficient data for learning.

[0148] The consumer thread consumes experience data from the buffer and performs network parameter update and optimization steps in response to the shared circular experience buffer reaching a preset threshold or a timer timeout event.

[0149] Among them, "filling status reaches the preset threshold" means that the number of valid experience data entries stored in the circular buffer exceeds a certain proportion (such as 80%) of the total capacity. "Timer timeout event" refers to an event signal emitted by an independent timer when it reaches a preset time (such as every 5 seconds), which is used to periodically trigger learning even when data generation is slow.

[0150] Specifically, the consumer thread is typically in a blocked waiting state. It listens for two conditions: the fill rate of the circular buffer and a periodic timer. When either condition is met, the thread is awakened. Upon awakening, the thread reads (consumes) a certain amount of the latest experience data from the circular buffer in batches, and then executes the network parameter update and optimization process defined in step S150. After completion, the thread returns to the blocked waiting state. For example, the consumer thread is configured to be awakened when the circular buffer fill rate exceeds 80% or every 5 seconds. Assuming the buffer capacity is 5000, when 4001 experiences are stored, the fill rate is 80.02%, triggering the consumer thread. After being awakened, the thread copies the most recent 2000 experiences from the buffer to its local memory and then begins network training based on this data, a process that may last 300ms. After training is complete, the thread sleeps, waiting for the next trigger.

[0151] Deploy a parameter server to store and provide multiple versioned copies of the policy network and value evaluation network parameters of the reinforcement learning agent.

[0152] The parameter server is a centralized, thread-safe parameter storage and management service. It stores the latest values ​​of all weight and bias parameters of the neural network and assigns them version numbers to support multiple threads safely reading and updating parameters. Versioned copies mean that the parameter server not only saves the latest parameters but may also retain snapshots of historical parameters from the most recent updates, facilitating rollback or comparison.

[0153] Specifically, the system can allocate a mutex-protected data structure in shared memory to store the parameter tensors of the policy network and the value evaluation network, along with a corresponding version number (such as an incrementing integer). This data structure provides two atomic operations: read (version number) and write (new parameter, new version number). For example, the parameter server is implemented as a simple class containing a dictionary `parameters` (keys are network names, values ​​are parameter tensors and version numbers) and a mutex `lock`. The initial version number is 1, storing randomly initialized network parameters. When the producer thread needs parameters, it calls the `get_parameters("policy", version)` method, and the parameter server returns a copy of the policy network parameters corresponding to the version number `version`.

[0154] The producer thread is configured to read the latest copy of the policy network parameters from the parameter server at the beginning of each control cycle for policy execution in the current cycle; the consumer thread is configured to publish the updated network parameters to the parameter server atomically after completing the network parameter update.

[0155] Specifically, at the beginning of each control cycle of the producer thread, before performing any decision-related calculations, the thread initiates a read request to the parameter server. The parameter server returns the latest version of the policy network parameters, which the producer thread loads into its local policy network model for state observation vector processing and forward inference within this cycle. After completing a network parameter update (i.e., step S150), the consumer thread packages the updated policy network and value evaluation network parameters, generates a new version number (e.g., the old version number + 1), and then calls the parameter server's write interface. This write operation is atomic, meaning that while the parameter server is updating its internally stored parameters and version number, read and write requests from other threads are blocked to ensure data consistency. For example, at the beginning of the cycle, the producer thread reads policy network parameters with version number v=105 from the parameter server. It uses this set of parameters for decision-making throughout the cycle. Meanwhile, the consumer thread completes a training cycle and generates new parameters. It submits the new parameters and version number v=106 to the parameter server. Within an atomic operation, the parameter server overwrites the old parameters with the new ones and updates the version number to 106. In the next control cycle, the producer thread reads the new parameters, version 106.

[0156] Therefore, according to the above implementation method, the system can completely decouple the flight control loop, which has extremely high real-time requirements, from the computationally intensive and time-consuming model learning loop through a clear producer-consumer architecture. The producer thread focuses on ensuring the real-time, stable, and safe operation of the control, unaffected by fluctuations in the learning process; the consumer thread utilizes the system's idle computing resources to perform deep model optimization at its own pace. The parameter server, acting as a secure and efficient data bridge between the two, ensures that the newly learned knowledge can be applied to online control in a timely and consistent manner. This architecture ultimately achieves parallel operation of flight control and online learning, enabling the UAV to continuously learn and evolve its control strategy from accumulated flight experience while performing uninterrupted inspection tasks, truly embodying the core value of online adaptive optimization.

[0157] Figure 3 This is a structural block diagram of a main distribution network UAV flight path correction system according to an embodiment of the present invention.

[0158] like Figure 3 As shown, the main power grid UAV flight path correction system includes: The observation vector generation module 210 is used to collect real-time sensor data from the UAV and compare it with a preset reference path model to generate a state observation vector.

[0159] The control command output module 220 is used to input the state observation vector into the policy network of the reinforcement learning agent and output high-level control commands.

[0160] The control switching module 230 is used to send high-level control commands to the low-level flight control system for execution. At the same time, an independent safety monitoring module monitors the flight status in real time and triggers control switching when the status exceeds the limit.

[0161] The experience data caching module 240 is used to collect new environmental states and calculate rewards based on reward functions when the UAV changes state, and store the experience data including state, action, reward and next state into the experience playback buffer.

[0162] The network parameter update module 250 is used to sample historical experience data in the experience replay buffer and update the network parameters of the reinforcement learning agent.

[0163] The flight path correction module 260 is used to cyclically execute the steps of generating state observation vectors, issuing instructions and monitoring safety, and storing experience during the execution of flight missions, and to asynchronously execute the step of updating network parameters.

[0164] The specific functions and examples of each module and submodule of the device in this embodiment of the invention can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0165] According to embodiments of the present invention, the above-described method of the present invention can be applied to a computer device and a readable storage medium.

[0166] Figure 4 A schematic block diagram of a computer device 600 that can be used to implement embodiments of the present invention is shown. The computer device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The computer device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0167] like Figure 4As shown, the computer device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the computer device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0168] Multiple components in computer device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows computer device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0169] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as a main distribution network UAV line-following flight path correction method. For example, in some embodiments, a main distribution network UAV line-following flight path correction method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the computer device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the main distribution network UAV line-following flight path correction method described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured by any other suitable means (e.g., by means of firmware) to perform a main distribution network UAV line-following flight path correction method.

[0170] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0171] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0172] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0174] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0175] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0176] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0177] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for correcting the flight path of a main distribution network UAV following a linear trajectory, characterized in that, include: Collect real-time sensor data from the UAV and compare it with a preset reference path model to generate a state observation vector; The state observation vector is input into the policy network of the reinforcement learning agent to output high-level control commands. The high-level control commands are sent to the low-level flight control system for execution, while an independent safety monitoring module monitors the flight status in real time and triggers a switch of control when the status exceeds the limit. In response to the drone's state transition, the new environmental state is collected and the reward is calculated based on the reward function. The experience data, including the state, action, reward and the next state, is stored in the experience replay buffer. Historical experience data is sampled from the experience replay buffer to update the network parameters of the reinforcement learning agent; During the flight mission, the steps of generating the state observation vector, issuing commands and monitoring safety, and storing experience are executed cyclically, and the steps of updating the network parameters are executed asynchronously with the above steps.

2. The method according to claim 1, characterized in that, The process involves collecting real-time sensor data from the UAV and comparing it with a preset reference path model to generate a state observation vector, including: By fusing satellite positioning signals with inertial measurement unit data, the attitude information of the UAV is obtained; Environmental image data is acquired using a visual sensor; Based on the pose information, the model is registered with the reference path model containing the spatial topology of the route, and the pose deviation of the UAV relative to the reference path and the rate of change of the pose deviation are calculated. Based on the environmental image data, the geometric look-ahead features of the path are extracted. These geometric look-ahead features are used to characterize the curvature properties of the path and the changing trend of these curvature properties. The environmental image data is input into a pre-trained deep neural network to extract environmental threat prospective features, which include the category, location, and presence confidence information of targets within the route corridor. The pose deviation, the rate of change of the pose deviation, the geometric look-ahead feature, and the environmental threat look-ahead feature are combined to generate the state observation vector.

3. The method according to claim 1, characterized in that, The process of inputting the state observation vector into the policy network of the reinforcement learning agent and outputting high-level control instructions includes: The state observation vector is input into the policy network of the reinforcement learning agent, wherein the policy network is a multilayer perceptron or a recurrent neural network that includes an attention mechanism; The policy network processes the input state observation vector and outputs a normalized original action command, which includes the desired forward velocity component, lateral velocity component, vertical velocity component and yaw rate component. The feasible range of each action component is dynamically calculated based on the real-time status information of the UAV using the action mask module. The real-time status information includes remaining power, structural load, ambient wind speed, and distance to obstacles. Based on the feasible range calculated by the action mask module, the original action instructions output by the policy network are constrained to generate the higher-level control instructions.

4. The method according to claim 1, characterized in that, The process of sending the high-level control commands to the low-level flight control system for execution, while an independent safety monitoring module monitors the flight status in real time and triggers a switch of control when the status exceeds the limits, includes: A pre-configured security monitoring module with a hybrid architecture is provided, which integrates a rule-based runtime verification system and a learning-based anomaly detector. The rule-based runtime verification system performs continuous state verification on predefined multi-level security attributes, including airspace geofence, attitude angle envelope, velocity envelope, and sensor data validity. The learning-based anomaly detector is loaded with an autoencoder or a single-class support vector machine model, and a normal behavior benchmark is constructed based on the historical normal flight state sequence of the UAV. Anomaly pattern detection is then performed on the real-time flight state sequence. In response to an alarm triggered by the rule-based runtime verification system or the learning-based anomaly detector, a control switching operation is performed based on the threat level of the alarm. The control switching operation includes a smooth transition from the policy network of the reinforcement learning agent to a preset backup controller, or a switch to a preset emergency return trajectory tracking controller.

5. The method according to claim 1, characterized in that, The steps of collecting new environmental states, calculating rewards based on reward functions, and storing experience data including states, actions, rewards, and the next state into an experience replay buffer include: Based on the new environmental sensing data collected after the UAV completes the state transition, a state perception and feature extraction process is executed to generate a new state observation vector. A multi-objective reward function is invoked to evaluate the state transition process and calculate an immediate reward signal. The multi-objective reward function integrates event-driven sparse rewards and periodically provided dense rewards. The triggering conditions and reward types for the sparse reward include: generating a positive reward in response to a progress event of completing a preset distance of inspection tasks; and generating a negative reward in response to a collision or task timeout event. The dense reward is calculated in each control cycle. The dense reward includes: a tracking reward for incentivizing a reduction in pose deviation; a smoothing reward for incentivizing a smooth change in control commands; an efficiency reward for incentivizing an increase in cruise speed while meeting tracking accuracy; and a safety reward for incentivizing the maintenance of a safe distance from obstacles. When the distance between the UAV and the obstacle is less than the dynamic safe distance, the safety reward is penalized. The state observation vector before migration, the executed high-level control instructions, the calculated reward signal, and the new state observation vector are encapsulated into empirical data and written into the empirical playback buffer.

6. The method according to claim 1, characterized in that, The step of sampling historical experience data from the experience replay buffer and updating the network parameters of the reinforcement learning agent includes: Maintain a priority experience replay buffer, which is logically divided into a high-priority storage area and a normal storage area; Based on the time-series differential error corresponding to each piece of stored experience data, the priority score of the experience data is dynamically calculated and maintained, and the experience data is stored in the selected storage area according to the priority score. Based on the priority scores of the empirical data, a priority-based random sampling strategy is adopted to sample historical empirical data from the high-priority storage area with a higher probability than sampling from the ordinary storage area. Importance sampling weights are applied to the data sampled from the high-priority storage area; Offline policy optimization is performed based on the sampled historical experience data. The policy gradient is calculated to update the policy network parameters of the reinforcement learning agent, the value function gradient is calculated to update the value evaluation network parameters of the reinforcement learning agent, and the target network parameters corresponding to the policy network and the value evaluation network are updated synchronously based on a soft update policy.

7. The method according to claim 1, characterized in that, The loop executes the steps of generating the state observation vector, issuing instructions and performing security monitoring and experience storage, and asynchronously executes the step of updating the network parameters, including: Deploy a producer-consumer concurrent model, with the producer thread serving as a high-frequency real-time control thread and the consumer thread serving as a low-frequency offline learning thread; The producer thread performs the following actions in a fixed control cycle: generating the state observation vector, performing forward inference based on the current policy network parameters to output the higher-level control command, verifying the flight status of the independent safety monitoring module, and writing the generated experience data into a shared circular experience buffer, thus forming a control loop. In response to the shared circular experience buffer reaching a preset threshold or a timer timeout event, the consumer thread consumes experience data from the buffer and performs the network parameter update and optimization steps. Deploy a parameter server to store and provide multiple versioned copies of the policy network and value evaluation network parameters of the reinforcement learning agent; The producer thread is configured to read the latest copy of the policy network parameters from the parameter server at the beginning of each control cycle for policy execution in the current cycle; the consumer thread is configured to publish the updated network parameters to the parameter server atomically after completing the network parameter update.

8. A main distribution network UAV flight path correction system, characterized in that, include: The observation vector generation module is used to collect real-time sensor data from the UAV and compare it with a preset reference path model to generate state observation vectors. The control command output module is used to input the state observation vector into the policy network of the reinforcement learning agent and output high-level control commands. The control switching module is used to send the high-level control commands to the low-level flight control system for execution. At the same time, an independent safety monitoring module monitors the flight status in real time and triggers control switching when the status exceeds the limit. The experience data caching module is used to collect new environmental states and calculate rewards based on the reward function when the UAV changes state. It stores the experience data, including state, action, reward and next state, into the experience replay buffer. The network parameter update module is used to sample historical experience data in the experience replay buffer and update the network parameters of the reinforcement learning agent. The flight path correction module is used to cyclically execute the steps of generating the state observation vector, issuing instructions and monitoring safety, and storing experience during the flight mission, and to asynchronously execute the step of updating the network parameters.

9. A computer device, characterized in that, include: At least one processor; and a memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, in, Computer instructions are used to cause a computer to perform the method according to any one of claims 1-7.

Images