An unmanned aerial vehicle autonomous navigation system based on hierarchical reinforcement learning strategy

Abstract

The application discloses an unmanned aerial vehicle autonomous navigation system based on a layered reinforcement learning strategy, which is suitable for three-dimensional flight tasks in unknown environments. The system comprises a state perception module, a layered strategy network, a control execution module, a data classification module and a data playback module. The state perception module extracts obstacle position information based on a deep neural network, and generates a state vector and a time sequence by fusing target, obstacle position and flight state. The layered strategy network generates a navigation intention by using a high-level DQN, and outputs a continuous control action by combining a low-level LSTM and PPO. The control execution module adjusts the attitude of the unmanned aerial vehicle according to the instruction and performs closed-loop correction. The system introduces a double dynamic memory mechanism (DDM), improves the strategy training efficiency and stability by experience classification and proportional sampling, and guides the strategy to optimize and converge among task completion, obstacle avoidance safety and flight rationality by using a multi-target reward function. The system has good environmental adaptability and generalization ability, and is suitable for autonomous navigation tasks in complex scenes.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of unmanned aerial vehicle autonomous visual navigation, and particularly relates to an unmanned aerial vehicle autonomous navigation system based on a hierarchical reinforcement learning strategy. BACKGROUND

[0002] With the development of artificial intelligence and embedded control technology, the problem of autonomous navigation of unmanned aerial vehicles in unknown complex environments has gradually become a hot spot in research and engineering applications. Complex environments usually have characteristics such as closed structure, narrow space, and dense obstacles, and traditional path planning methods (such as A*, RRT, SLAM, etc.) are difficult to cope with dynamic obstacle changes and unknown environment exploration problems, and have limitations such as incoherent navigation path and untimely obstacle avoidance. In order to improve the flexibility and environmental adaptability of navigation, deep reinforcement learning has been widely used in unmanned aerial vehicle navigation control systems in recent years, which has the ability to learn optimal control strategies directly from the perception end and is suitable for high-dimensional continuous control and complex decision-making tasks.

[0003] In existing research, many methods attempt to introduce deep neural networks into the autonomous navigation task of unmanned aerial vehicles. For example, patent CN118963407A proposes a flight path planning method based on Q learning, which can learn a suitable path in a fixed obstacle environment. CN119131643A uses an image feature fusion method to improve obstacle detection. CN114212683A introduces a double neural network structure to realize target-oriented flight control. Although the above patents optimize the autonomous navigation task of unmanned aerial vehicles, they still rely on single-frame images for decision-making, lack the ability to model time dynamics, and have slow strategy convergence and limited generalization ability, resulting in uncoordinated responses to complex task objectives and action regulation.

[0004] In addition, existing reinforcement learning systems mostly use a unified experience pool replay strategy to store and sample all training experiences equally, which makes it difficult to highlight the training of key states or high-risk decision states, affecting the speed of strategy convergence and the stability of the effect. For example, some technologies indirectly guide the strategy to focus on the target state through the reward function setting, but do not effectively model and regulate the experience management level, limiting the effective optimization of complex navigation strategies.

[0005] In summary, the existing autonomous navigation methods of unmanned aerial vehicles in complex environments mainly have the following technical problems:

[0006] The control strategy structure is single, and lacks the ability to decouple task-level navigation instructions and execution-level actions;

[0007] The modeling of obstacle dynamic characteristics is insufficient, and the strategy lacks the ability to understand the time dependence;

[0008] The experience replay mechanism cannot classify and manage according to the state value and directional training, and the sample utilization efficiency is low.

[0009] In order to solve the above problems, the application provides an unmanned aerial vehicle autonomous navigation system based on a hierarchical reinforcement learning strategy, which introduces image perception, multi-modal state construction, hierarchical strategy structure and double dynamic memory mechanism (DDM) modules, improves the navigation adaptability, obstacle prediction ability and training efficiency of the system in unknown environments, and is suitable for high dynamic flight tasks in variable and dense obstacle environments. SUMMARY

[0010] The application provides an unmanned aerial vehicle autonomous navigation system based on a hierarchical reinforcement learning strategy, which combines deep visual perception, state time series modeling and high-low layer strategy decoupling control, realizes intelligent obstacle avoidance and target guidance of the unmanned aerial vehicle in unknown environments. The system uses a deep neural network for obstacle detection and state fusion, generates a navigation intention through a high-layer DQN, outputs a three-dimensional continuous control action through a low-layer LSTM and PPO, and combines a double dynamic playback mechanism to improve the strategy learning efficiency and robustness, and is suitable for complex, dynamic and multi-obstacle flight task scenarios.

[0011] The technical scheme adopted by the application is as follows:

[0012] An unmanned aerial vehicle autonomous navigation system based on a hierarchical reinforcement learning strategy, at least comprising:

[0013] A state perception module is used to collect flight environment images and extract obstacle relative position, direction angle and other information, simultaneously acquire target position and unmanned aerial vehicle flight state parameters, fuse multi-source perception data to construct a current state vector, and generate a state time series through a state cache for navigation strategy calculation and action control by a hierarchical strategy network module;

[0014] A hierarchical strategy network module is used to generate a flight control instruction based on the state vector or the state time series, and the hierarchical strategy network module comprises:

[0015] A high-layer strategy submodule is used to generate a navigation direction intention based on the current state vector, the navigation direction is a type in a predefined discrete direction set, and a deep Q learning network (DQN) is used for strategy optimization;

[0016] A low-layer strategy submodule is used to receive the navigation direction intention and the state time series, and generate a continuous action instruction through a neural network combining a long short-term memory network (LSTM) structure and a proximal policy optimization algorithm (PPO), the action instruction comprising a yaw angular velocity, a pitch angular velocity and a thrust value of the unmanned aerial vehicle;

[0017] The control execution module is configured to control the attitude adjustment and propulsion of the UAV according to the action instruction, and to perceive current flight state information in real time, including attitude angle, flight speed and flight height and the like, for closed-loop correction and experience sample construction.

[0018] The double dynamic memory mechanism (DDM) is configured to dynamically store and schedule flight experience samples, so as to improve sample utilization efficiency and convergence stability of strategy training, and is composed of a data classification module and a data playback module.

[0019] The data classification module is configured to divide experience data into three categories of result experience, dangerous experience and safe experience according to flight results.

[0020] The data playback module is configured to construct a double-mini-batch shunting structure based on a layer field in the sample, so as to optimize parameters of the high-level DQN network and the low-level LSTM+PPO strategy network respectively.

[0021] The overall system is shown in FIG. 1. Figure 1 As shown in FIG. 1, the UAV navigation system architecture for three-dimensional space and dynamic obstacle environment is constructed through the cooperation of the above modules, and the system has layered control capability, state sequence modeling capability and experience playback management capability, and can support the UAV to realize autonomous obstacle avoidance and navigation with robustness and responsiveness in an unknown environment.

[0022] According to the above scheme, the state perception module is composed of an image acquisition submodule, an obstacle detection submodule and a post-processing submodule, and has the functions of realizing visual perception of the environment in front of the UAV, extracting spatial position information of obstacles, fusing obstacle information and navigation target information output by the obstacle detection and post-processing submodules, combining the flight state of the UAV itself, constructing a structured current state vector, and further generating a state time sequence to provide a unified and standardized perception for the layered strategy network module.

[0023] The image acquisition submodule acquires flight environment images in real time through a visual sensor (such as an RGB camera, a depth camera or a binocular camera) mounted on the front end of the UAV body. In order to adapt to the subsequent target detection model, the image frame needs to be standardized and preprocessed before input, including:

[0024] Size unification: cropping the focal point area of the image and scaling to the network input size (such as 224x224 or 300x300 pixels);

[0025] Channel format conversion: converting the image channel from HWC format to CHW format required by the deep learning framework;

[0026] Pixel normalization: Standardize the image according to the statistical parameters of the dataset used for model training (such as the mean and standard deviation of ImageNet), to improve the generalization ability of the model under complex lighting and color conditions. The obstacle detection sub-module is used to receive the standardized preprocessed image frames, and based on a deep neural network, it realizes the automatic identification and positioning of potential obstacles in the image, and outputs the two-dimensional bounding box coordinates and corresponding class labels of the obstacles in the image plane;

[0027] Preferably, the obstacle detection module adopts a two-stage target detection structure based on Faster R-CNN, combined with a feature pyramid network (FPN) to improve the detection ability of obstacles of different scales. This module includes the following core components:

[0028] Feature extraction network: Preferably, ResNet-50 is used as the backbone network, combined with residual connection and downsampling operation to extract multi-level semantic features;

[0029] FPN multi-scale fusion: Through the transmission and fusion of high-level semantics to low-level features, multi-scale feature maps are generated to improve the detection accuracy of large and small targets;

[0030] Region proposal network (RPN): Generate anchor boxes on each scale feature map and output foreground classification scores and boundary regression information, and select candidate regions through NMS;

[0031] ROI alignment and refining module: Use ROI Align to extract candidate region features, perform classification and regression optimization, and output the final obstacle box and class.

[0032] This detection process uses a joint loss function to optimize the classification and regression subtasks, taking into account detection accuracy and boundary fitting effects, and is suitable for multi-class obstacle recognition in complex indoor scenes.

[0033] The post-processing module receives multiple target bounding boxes (bounding boxes) output from the obstacle detection network (Faster R-CNN), each box containing the upper left and lower right pixel coordinates [x1, y1, x2, y2]. The post-processing module first calculates the center point coordinates of each box:

[0034]

[0035] This center point represents the projection position of the obstacle in the image plane. According to the horizontal coordinate x c of the center point, the deviation angle θ of the obstacle relative to the front view of the UAV is estimated in the image width W. Assuming that the horizontal field of view angle of the camera is α (such as ±60°, i.e. total field of view is 120°), the direction angle calculation formula is:

[0036]

[0037] The angle value represents the degree of left or right deviation of the obstacle relative to the front of the flight, and a negative value indicates deviation to the left and a positive value indicates deviation to the right. For the vertical size (height) or area of the detection frame, the relative distance d between the obstacle and the unmanned aerial vehicle is estimated based on the following principle:

[0038]

[0039] where h = y2 - y1 represents the height of the bounding box, and k is a proportionality constant obtained by presetting or fitting based on the camera focal length and the actual size of the obstacle, for establishing the mapping relationship between the pixel scale and the physical distance. For obstacles with known structures (such as doors and columns), the relative distance can be inversely deduced based on the prior scale. If the system is equipped with a depth camera, the average depth value of the region in the depth map can be directly obtained as d. Then all the detected obstacles are sorted by distance from near to far, and at most three targets are retained, each target is encoded in polar coordinates as (θ i ,d i ), and finally a fixed-length state vector is formed as follows, which is the input of the state construction module:

[0040] s obstacle =[θ1,d1,θ2,d2,θ3,d3]

[0041] If the detection result is less than three targets, the default item is filled with the maximum distance value and 0° angle (or specified default value).

[0042] The state perception module constructs the relative position information (direction angle, distance) of the navigation target, the polar coordinate information of the obstacle output by the post-processing (up to 3 obstacle (θ i ,d i )) and the current flight state information of the unmanned aerial vehicle (speed, heading angle, climb angle, flight height, etc.) into a structured current state vector, and further generates a state time series to provide a unified and standardized perception input for the hierarchical policy network module.

[0043] The current state vector is a fixed-length real vector that collects the above-mentioned multi-source perception information, totaling 15 dimensions, and is expressed in the following form:

[0044] s t =[θ g ,d g ,Δh t ,Δh min ,θ1,d1,θ2,d2,θ3,d3,v x ,v y ,v z ,ψ t ,γ t ]

[0045] wherein:

[0046] θ g : target direction angle, i.e., the deflection angle of the navigation target relative to the current orientation of the UAV;

[0047] d g : target distance, i.e., the Euclidean distance between the current position of the UAV and the navigation target;

[0048] Δh t : the difference between the current flight height and the target height;

[0049] Δh min : the difference between the current flight height and the set minimum flight height;

[0050] (θ i ,d i ) represents the relative direction angle and distance of the i-th obstacle;

[0051] v x ,v y ,v z : respectively represent the linear velocity components of the UAV in the three-axis direction, reflecting its motion dynamic information;

[0052] ψ t : current heading angle;

[0053] γ t : current climb angle.

[0054] To meet the input specification of the hierarchical policy network, improve numerical stability and policy convergence speed, the state construction submodule normalizes all input features, mainly including:

[0055] Distance type features (such as d g ,d i ) are normalized to the [0, 1] interval according to the maximum perception distance;

[0056] Angle type features (such as θ g ,θ i ,ψ t ) are standardized to [-1, 1] to adapt to the periodicity of angles;

[0057] Dynamic state type features (such as Δh t ,,Δh min ,v x ,v y ,v z ) are mapped to dimensionless values with reference to the performance parameter range of the aircraft.

[0058] In addition, in order to enhance the modeling ability of the strategy on time dependence, the module introduces a sliding cache mechanism to record the recent continuous T frame state vectors to form a state time sequence:

[0059] S t =[s t-T+1 ,s t-T+2 ,...,s t ]

[0060] The state time sequence is transmitted as input to a long short-term memory network (LSTM) in a low-level policy sub-module for capturing the trend features of the evolution of the obstacle position and flight state over time. The LSTM network has the ability to model long-term dependencies and can implicitly learn dynamic change features without explicitly predicting the future state of the obstacle, generate smoother and more responsive continuous control actions, and effectively improve the navigation accuracy and obstacle avoidance robustness of the system in a dynamic unknown environment.

[0061] According to the above scheme, the hierarchical policy network module is constructed based on a deep reinforcement learning framework and is divided into two functional sub-units, a high-level policy sub-module and a low-level policy sub-module, as shown in Figure 2 and Figure 3 Through module decoupling and task layering, the high-dimensional control requirements in a complex environment are effectively addressed, and the reaction speed and obstacle avoidance accuracy of the unmanned aerial vehicle in the navigation task are improved.

[0062] Specifically, the high-level policy sub-module is implemented based on a deep Q-network (DQN) structure. This algorithm combines Q-learning and a deep neural network to estimate the state-action value (Q value) of each candidate navigation direction in a discrete action space. Through continuous sampling of environmental feedback and cumulative rewards, DQN can learn to choose the optimal navigation direction under a given state, thereby realizing goal-oriented policy optimization.

[0063] In the system, the DQN inputs the current state vector and outputs the Q value vector of each navigation direction. To balance policy exploration and utilization, the invention preferably uses an epsilon-greedy policy for action selection, i.e., randomly selecting any navigation direction with a probability of epsilon to enhance the exploration ability of the state space, and selecting the navigation direction with the maximum Q value with a probability of 1-epsilon to realize the gradual convergence of the optimal policy. The epsilon value can be gradually decayed, with a larger epsilon (such as 0.9) set in the early training stage to increase exploration, and gradually reduced to a smaller value (such as 0.05) in the later training stage to improve the stability of the policy.

[0064] The core function of the high-level policy submodule is to output the global navigation direction intention according to the current state vector. The current state vector is input into the DQN network for navigation decision making. The high-level policy submodule outputs the Q value of each candidate navigation direction according to the state vector, and selects the optimal navigation direction according to the ε-greedy strategy.

[0065] In the preferred embodiment, the DQN network structure includes three hidden layers (fully connected network), with neuron numbers of 128, 256 and 128 in sequence, all using ReLU activation function, and the output layer being a linear layer with an output dimension of 8 (i.e. Q values of 8 navigation directions). The predefined navigation direction set is:

[0066] A = {0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°}.

[0067] There are a total of 8 categories, each representing a typical flight direction (e.g. front, left front, right, back, etc. eight standard flight directions). The DQN outputs a number a high ∈ {1, 2,..., 8} representing the optimal navigation direction intention under the current state, and the number is transmitted to the low-level policy submodule as prior guidance. Alternatively, the direction number can also be converted into a unit vector form, such as

[0068] Table 1, for vector comparison with the current heading or as a strategy guidance input.

[0069]

[0070] Table 1

[0071] In this system, the high-level policy submodule uses DQN network for discrete action policy learning, and the optimization goal is to minimize the temporal difference (TD) error of the state-action value function. The loss function is defined as follows:

[0072]

[0073] Wherein, the target Q value is defined as the weighted sum of the actual reward and the maximum value of the next state Q value:

[0074]

[0075] The variables are defined as follows:

[0076] s t represents the current state vector (15 dimensions), which is composed of target information, obstacle polar coordinates and unmanned aerial vehicle state;

[0077] a trepresents the current selected discrete navigation direction index (8 classes in total);

[0078] r t represents the current reward, based on the flight performance feedback (e.g. target approaching, obstacle avoidance success, collision penalty, etc.);

[0079] s t+1 represents the next time step state vector, used for Bootstrap evaluation;

[0080] Q(s t ,a t ; θ) represents the Q value estimation of the current policy network for the state-action pair;

[0081] θ represents the current parameters of the Q network;

[0082] θ - represents the target Q network with delayed synchronization, used for stable training;

[0083] γ represents the discount factor, usually set to 0.99;

[0084] a' represents the possible action in the next state s t+1 .

[0085] D high represents the mini-batch sampled only from high layer samples with layer = high.

[0086] (1-done t ) term is used to mask the future reward term when the episode is terminated.

[0087] To support the training of the policy network, the system uses a structured experience sample form. Each experience data is stored in the form of an eight-tuple:

[0088] ε t = {s t ,a t ,r t ,s t+1 ,done t ,int t ,type, layer}

[0089] Where:

[0090] s t represents the current state vector, including target information, obstacle polar coordinates, aircraft state, etc., used for network input;

[0091] a t represents the current action, high layer is discrete direction, low layer is continuous control action (yaw angular velocity, pitch, thrust);

[0092] r t represents the current reward, used to evaluate the pros and cons of the current action, and is the direct feedback of policy optimization;

[0093] s t+1 represents the next state, used to calculate the TD error (DQN) or advantage function (PPO);

[0094] done t ∈{0,1}, indicates whether to terminate (such as task success / collision);

[0095] int t represents the navigation intention, which is the high-level output from DQN, and is used as the context input of the low-level control policy;

[0096] type represents the experience type RE (result experience), DE (danger experience), SE (safe experience), which is used to classify sampling

[0097] layer represents the level to which it belongs, "high" (high-level policy sample) or "low" (low-level policy sample), which is used for hierarchical playback;

[0098] The high-level policy submodule (DQN) is trained only using the five-tuple composed of the first five items:

[0099]

[0100] The low-level policy submodule (LSTM+PPO) uses the complete eight-tuple to perform trajectory sampling and policy optimization:

[0101]

[0102] The entire trajectory form (episode):

[0103]

[0104] In the system, the low-level policy submodule is used to generate continuous action control instructions to realize dynamic adjustment of the UAV attitude and propulsion. This module receives the navigation direction intention a hign from the high-level policy submodule and the state time sequence S t generated by the state construction module. The state sequence is input into the long short-term memory network (LSTM) to extract the trend of obstacle direction and distance, the evolution law of flight attitude, and other time sequence features. The LSTM network has the ability to model historical dependencies and can capture environmental dynamic evolution characteristics without explicitly predicting future states of obstacles, thereby assisting the policy network to generate control instructions with smoothness and advance response capability;

[0105] After processing, the LSTM outputs the hidden state vector h(t) of the last time step, which encodes the obstacle changes and flight dynamic patterns in the near period as a temporal context feature. This context feature and the navigation direction intention a hign (One-Hot encoding or direction vector) together constitute the strategy network input, which is transmitted into the continuous action strategy network trained based on the Proximal Policy Optimization (PPO) algorithm;

[0106] The PPO strategy network is a policy gradient optimization method in reinforcement learning, which has the advantages of fast convergence speed, strong training stability, and adaptation to continuous action space, and is suitable for multi-dimensional dynamic control tasks. Its optimization objective function is as follows:

[0107]

[0108] Among them:

[0109] r t (θ) represents the probability ratio of the current policy to the old policy;

[0110] θ represents the parameters of the current policy network;

[0111] represents the advantage function, which measures the superiority of the current action relative to the average policy;

[0112] ∈ represents the clipping coefficient, which controls the policy update amplitude, usually set to 0.1-0.2;

[0113] clip(·) represents clipping the probability ratio to the range of [1-∈, 1+∈], to prevent performance degradation caused by excessive policy changes;

[0114] π θ (a t ,s t ) represents the probability of taking action a t under the current policy;

[0115] π old (a t ,s t ) represents the same action probability under the old policy, as a reference baseline.

[0116] In the preferred embodiment, the low-level strategy network structure includes a Long Short-Term Memory network (LSTM) and a policy network based on the Proximal Policy Optimization algorithm (PPO), wherein:

[0117] The LSTM network includes one LSTM layer with 128 neurons, which outputs a context vector h(t) for characterizing the state time series features.

[0118] The PPO policy network adopts a double-branch structure based on an Actor-Critic architecture:

[0119] Both the Actor branch and the Critic branch are fully connected neural networks, each containing two hidden layers with neuron counts of 64 and 32, respectively, and using a ReLU activation function.

[0120] The Actor network takes the context vector h(t) output by the LSTM and the high-level navigation intention as input, and outputs a continuous three-dimensional action control vector a(t), represented as:

[0121] a(t) = [ω ψ ,ω γ , T]

[0122] ω ψ : controls the yaw angular velocity of the UAV around the vertical axis, used to realize heading adjustment;

[0123] ω γ : controls the pitch angle change rate of the UAV, used to realize the tilt of ascending or descending;

[0124] T: thrust control, used to adjust the flight propulsion speed or maintain lift balance.

[0125] This continuous action vector is used to control the heading, climb angle, and propulsion state of the UAV in three-dimensional space, supporting fine obstacle avoidance and dynamic response control.

[0126] The Critic network outputs the state value function V(s t ), which is used to evaluate the value of the current state and assist in optimizing the Actor network; the PPO network uses a clipping objective function for policy optimization to improve training stability and enhance policy generalization ability, where V(s t ) is:

[0127]

[0128] s t : current state;

[0129] π: the current executed policy (i.e., the rule for selecting actions);

[0130] r t+k : immediate reward from time t to k steps in the future;

[0131] γ∈(0,1]: discount factor, measures the importance of future rewards (the smaller the more "short-sighted").

[0132] In the system operation, the high-level strategy submodule periodically updates the navigation intention at a lower frequency (such as 10Hz), ensuring that the flight path has directional consistency in the macroscopic level; the low-level strategy submodule generates flight action instructions in real time at a higher frequency (such as 50Hz), improving the reaction ability to local environmental changes, and realizing the hierarchical modeling and dynamic execution of strategies across time scales.

[0133] In summary, the hierarchical strategy structure includes:

[0134] High-level strategy network (based on DQN): used to evaluate the value of each navigation direction under the current state and output discrete navigation intention;

[0135] Low-level strategy network (based on LSTM+PPO): used to model the time sequence state features and output continuous control actions;

[0136] This structure effectively improves the system's ability to respond to complex unknown environments and control accuracy, alleviates the training shock, navigation failure and obstacle avoidance failure problems faced by traditional single-layer reinforcement learning in high-dimensional action space, and is particularly suitable for complex autonomous flight task scenarios with dense obstacles and frequent state changes.

[0137] The present application further designs a structured reward function mechanism for guiding strategy optimization, which is used to quantify the pros and cons of the current action during training. The reward function mainly consists of three parts: first, the arrival target reward, which gives positive encouragement when the UAV successfully reaches the navigation target area; second, the collision penalty, which applies negative feedback when obstacles are contacted or the forbidden flight area is invaded; third, the boundary penalty, which punishes the aircraft when it deviates from the task boundary. Through the weighted fusion of the above feedback items, the system can achieve the optimization goal of "encouraging task completion and punishing high-risk behavior", guiding the strategy to achieve a good balance between obstacle avoidance safety and navigation efficiency, thereby improving the convergence speed and strategy robustness of reinforcement learning training:

[0138] Arrival target reward r a :

[0139]

[0140] Collision penalty r c :

[0141]

[0142] Boundary penalty r o :

[0143]

[0144] Overall reward function expression:

[0145] r t= w1 · t a + w2 · r c + w3 · r o

[0146] wherein w1, w2, w3 are weight coefficients of each term, which are set as 1 by default and can be dynamically adjusted during the training phase.

[0147] According to the above scheme, the control execution module is configured to receive the three-dimensional continuous action vector a(t) output by the low-level policy submodule and convert it into a control instruction executable by the flight control system to drive the UAV to complete attitude adjustment and propulsion control in the three-dimensional space, as shown in FIG. 2. Figure 4 The control execution module analyzes and encapsulates the action vector to generate a data frame format compatible with the flight control system, and sends it to the flight control platform through the communication interface module to realize physical execution of the control instruction.

[0148] This module supports the following two deployment modes:

[0149] Online flight control mode: action instructions are sent to the physical flight platform (such as the PX4 system) in real time via the MAVLink communication protocol, and the ROS / MAVROS framework is used to communicate with the flight control firmware to directly control the attitude and speed of the UAV to achieve autonomous navigation and obstacle avoidance control.

[0150] Offline simulation deployment mode: the control execution module is connected to a virtual simulation platform (such as Gazebo or Microsoft AirSim) to drive the UAV model to perform motion control in the simulation environment, realizing training sample collection, policy testing, and task reproduction.

[0151] To improve system stability and closed-loop response capability, the control execution module is preferably connected to the flight state perception module through a data interface to receive real-time feedback information of key states of the UAV, including attitude angles (pitch, roll, yaw), flight speed, flight height, and other parameters. The flight state information can be used to:

[0152] dynamically correct the current action instruction to improve control accuracy;

[0153] detect and control flight abnormalities (such as height overrun, attitude imbalance, speed mutation, etc.) quickly and safely;

[0154] construct the flight feedback field in the training sample to enrich the data structure of the experience replay module.

[0155] In the preferred embodiment, the control execution module further integrates a safety control submodule for triggering emergency control instructions (such as hovering, emergency stop, return, etc.) when detecting abnormal yaw angle, too close distance to obstacles, or control instability, etc., to realize closed-loop safety protection and fault-tolerant processing of flight control, and improve the robustness and task safety of system operation.

[0156] As the end execution unit of the hierarchical strategy structure, the module cooperates with the low-level strategy submodule at a high frequency to support real-time response to continuous control instructions at a frequency of 50 Hz or higher, ensuring that the strategy output can be quickly executed to meet the comprehensive requirements for response delay, dynamic accuracy, and robustness of the control system in complex scenarios.

[0157] In summary, the control execution module not only undertakes the conversion and execution of flight instructions, but also realizes closed-loop control enhancement and data recording with flight state perception feedback, providing support and protection for the stability, response capability, and training effectiveness of the overall system.

[0158] According to the above scheme, the data classification module is used to classify and manage the experience data generated by the unmanned aerial vehicle in real time during the training process, providing structured support for subsequent experience playback and strategy optimization. This module constitutes the core basis of the Double Dynamic Memory (DDM) mechanism in the system, and its main function is to classify experience samples according to training value and risk level based on flight state and interaction results, in order to improve sample utilization efficiency and accelerate strategy convergence.

[0159] In the preferred embodiment, the experience data is divided into three categories according to the following criteria:

[0160] Result Experience (RE): refers to the complete state-action-reward sequence in which the unmanned aerial vehicle successfully reaches the target point or collides and terminates during flight, reflecting the overall performance of the current strategy in terms of task completion;

[0161] Dangerous Experience (DE): refers to a state segment in which the minimum distance between the unmanned aerial vehicle and obstacles is less than a preset safety threshold (such as 0.5 m) during flight, used to depict the behavior stability of the strategy in potentially high-risk scenarios;

[0162] Safe Experience (SE): refers to a state sequence in which the unmanned aerial vehicle does not trigger a collision and always maintains a safe distance from obstacles during flight, representing the running characteristics of the strategy in normal state;

[0163] The module dynamically evaluates the UAV state at each time step and divides the experience samples into corresponding experience pools in real time according to preset rules. To improve the retention probability of key experiences, the system sets different categories of storage weights, and the preferred configuration is shown in Table 2:

[0164]

[0165] Table 2

[0166] According to the above scheme, the data playback module is used to extract training samples from the three classified experience pools to support the parameter optimization process of the hierarchical policy network. Based on the "experience category-based structured sampling strategy" principle, the module respectively performs hierarchical scheduling on the RE, DE, and SE experiences in a set proportion to construct a training set with strong representativeness and reasonable distribution, thereby improving training efficiency and policy robustness.

[0167] In the preferred embodiment, the data playback module uses a fixed proportion sampling strategy (such as RE:DE:SE=60:30:10) based on the classified experience pool to construct the mini-batch used for training, wherein:

[0168] Resulting experience (RE): learning of the reinforcement strategy for successful paths and failure boundaries;

[0169] Dangerous experience (DE): improvement of the coping ability of the strategy in the critical high-risk state;

[0170] Safe experience (SE): enhancement of the stability and generalization performance of the strategy.

[0171] After sampling, the module further divides the uniformly sampled mini-batch into two subsets according to the layer field (marked as high or low) in each experience sample: discrete control samples (layer=high) for high-level DQN network training and continuous control samples (layer=low) for low-level LSTM+PPO network training. The two mini-batches are input into the respective policy networks in parallel, performing forward inference and gradient update, thereby realizing efficient collaborative optimization of the hierarchical policy structure.

[0172] According to the above scheme, the DDM (Double Dynamic Memory) mechanism is composed of the data classification module and the data playback module, as shown in Figure 5 which has the following two levels of dynamic processing capabilities:

[0173] Dynamic classification storage (first layer dynamic): during the strategy training process, the system classifies experience samples in real time based on flight state parameters, obstacle relative distance and navigation results, and divides them into three categories: result experience (RE), danger experience (DE) and safe experience (SE), and respectively saves them in the corresponding hierarchical experience pool according to the set storage probability (such as P RE = 0.95, P DE = 0.1, P SE = 0.01), thereby significantly improving the retention rate of key experience and reducing the interference of redundant samples on training efficiency and strategy stability.

[0174] Category-based sampling control (second layer dynamic): in the strategy optimization stage, the system constructs a training sample set from the three experience pools according to the target of different training stages, and combines the layer field marked in each sample to automatically divide the samples into two subsets, and constructs the double mini-batch shunt structure required by the high-layer DQN and low-layer LSTM+PPO strategy network to realize asynchronous parallel training and differential optimization of the hierarchical strategy network.

[0175] Compared with the indiscriminate random replay mechanism of the traditional unified experience pool, the DDM mechanism proposed in the application significantly improves the use efficiency and representativeness of experience samples through the three-dimensional structured design of "type-based storage + weight-based sampling + hierarchical parallel training", and strengthens the perception ability and response robustness of the strategy to key states in complex environments. It is especially suitable for high-dynamic, high-risk and information-incomplete three-dimensional autonomous flight task scenarios.

[0176] The application has the following beneficial effects:

[0177] A hierarchical reinforcement learning strategy architecture is proposed to construct a high-low decoupled control strategy network, wherein the high layer realizes global planning of navigation direction intention based on DQN, and the low layer realizes continuous action generation of three-dimensional attitude and propulsion control by combining LSTM and PPO, which balances the global guidance and local responsiveness of the strategy, and improves the navigation response speed and control accuracy;

[0178] A double dynamic memory mechanism (DDM) and a double mini-batch shunt structure are proposed, which classifies and stores experience samples and dynamically samples according to the training stage, and shunts them to high-low layer strategy networks for asynchronous training according to the layer field, thereby improving the key utilization efficiency of training samples and the learning stability of the strategy network;

[0179] The state perception module fuses target, obstacle and aircraft state information, and simultaneously introduces state time series and LSTM network modeling to strengthen the prediction ability and stable control ability of the strategy to dynamic scenes.

[0180] The control execution module supports MAVLink communication and simulation interfaces (such as Gazebo), facilitates pre-training of the algorithm in a simulation environment, and can be seamlessly migrated to an actual flight platform, and has good engineering realizability and cross-platform versatility;

[0181] The system integrates flight state feedback and safety policy judgment mechanism, and can automatically trigger emergency stop, hovering and other actions in abnormal state, realizes closed-loop safety control, and protects stable flight of the unmanned aerial vehicle in complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0182] The application will be further described below in combination with the drawings and examples, and the drawings are as follows:

[0183] Figure 1 Figure 1 is a general structure diagram of an unmanned aerial vehicle autonomous navigation system based on a hierarchical reinforcement learning strategy

[0184] Figure 2 Figure 3 is a high-level policy network flow chart

[0185] Figure 3 Figure 4 is a low-level policy network structure diagram

[0186] Figure 4 Figure 5 is a control instruction execution flow chart

[0187] Figure 5 Figure 6 is a structure diagram of a double dynamic memory mechanism DETAILED DESCRIPTION

[0188] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application, and are not used to limit the application.

[0189] Referring to Figures 1-5 An unmanned aerial vehicle autonomous navigation system based on a hierarchical reinforcement learning strategy, at least comprising a state perception module, a hierarchical policy network module, a control execution module, a data classification module and a data playback module. The specific steps are as follows:

[0190] S1: task modeling and state construction

[0191] The three-dimensional navigation task of the unmanned aerial vehicle is modeled as a Markov decision process (MDP), and the state space, action space and reward function are defined;

[0192] The system obtains environment images from the image acquisition device in real time, extracts the relative position of the obstacle through the image perception module (Faster R-CNN+FPN);

[0193] The flight control system provides the unmanned aerial vehicle attitude angle, flight speed and height information in real time;

[0194] The state construction module fuses the target position information, obstacle information, and flight state to construct a 15-dimensional state vector and generates a state time sequence as network input through a sliding buffer.

[0195] S2: System initialization

[0196] Initialize the simulation environment or real flight platform.

[0197] Set the initial UAV pose, target point position, and navigation area boundary.

[0198] Load the hierarchical policy network structure, including:

[0199] High-level DQN network structure and parameters;

[0200] Low-level LSTM+PPO network structure and parameters;

[0201] If there is a pre-trained model available, it is preferred to use; otherwise, use random initialization.

[0202] S3: Execute hierarchical policy control

[0203] The high-level policy network receives the state vector s t at a typical frequency of 10 Hz and outputs the navigation intention a high , providing directional guidance for the low-level;

[0204] The low-level policy network receives the navigation intention a high and the state time sequence S t as input to the policy network and outputs the continuous action vector a(t) = (ω ψ , ω γ , T);

[0205] The control execution module translates a(t) into control instructions and sends them to the flight control system through the MAVLink protocol for execution by the UAV.

[0206] S4: Safety protection mechanism

[0207] The control execution module synchronously receives IMU, radar, altimeter, and other flight state data;

[0208] If the system detects attitude instability, too close obstacles, or navigation abnormalities, it automatically triggers the safety submodule to perform emergency actions (such as hovering, emergency stop, or return);

[0209] Achieve closed-loop flight control and safety assurance.

[0210] S5: Experience collection and classification

[0211] Generate an experience eight-tuple εt = {s t , a t , r t , s t+1 , done t , int t , type, layer};

[0212] Data classification module divides samples into three categories: result experience (RE), danger experience (DE), and safe experience (SE) according to flight states.

[0213] Result experience (RE): navigation success or collision;

[0214] Danger experience (DE): close to obstacles;

[0215] Safe experience (SE): normal path;

[0216] Each category of experience is written into the experience pool with a set probability, constituting the first layer of the double dynamic memory mechanism (DDM).

[0217] S6: mini-batch construction and strategy optimization

[0218] The data playback module samples samples from the three categories of experience in proportion (e.g., RE:DE:SE = 60:30:10);

[0219] According to the layer field in the experience, the samples are automatically divided into two mini-batches:

[0220] The high-level DQN network optimization uses RE+DE discrete samples;

[0221] The low-level LSTM+PPO network uses SE samples and time series samples;

[0222] The high-level DQN uses the time series difference error TD-error as the loss function;

[0223] The low-level PPO network uses the clipping objective function for updating to improve the stability of the strategy.

[0224] S7: task termination and reset

[0225] If the target point is reached or an unrecoverable collision / heading out-of-bounds event occurs, the episode is terminated;

[0226] The system records the complete trajectory;

[0227] The aircraft state, target position, and cache are automatically reset, and the next training / execution cycle is entered.

[0228] It is to be understood that all such modifications and variations that can occur to those skilled in the art in the light of the foregoing description are to be considered within the scope of the application as defined in the claims appended hereto.

Claims

1. An unmanned aerial vehicle autonomous navigation system based on hierarchical reinforcement learning policy, characterized in that, The system comprises at least: a state perception module for collecting flight environment images and extracting obstacle relative position and direction angle information, obtaining target position and unmanned aerial vehicle flight state parameters, fusing multi-source perception data to construct a current state vector, and generating a state time sequence through a state cache for navigation strategy calculation and action control by a hierarchical strategy network module; a hierarchical strategy network module for generating flight control instructions based on the state vector or state time sequence, the hierarchical strategy network module comprising: a high-level strategy submodule for generating a navigation direction intention based on the current state vector, the navigation direction being a type in a predefined discrete direction set, and using a deep Q learning network (DQN) for strategy optimization; a low-level strategy submodule for receiving the navigation direction intention and state time sequence and generating continuous action instructions through a neural network combining a long short-term memory (LSTM) network structure and a proximal policy optimization (PPO) algorithm, the action instructions including yaw angular velocity, pitch angular velocity, and thrust value of the unmanned aerial vehicle; a control execution module for controlling attitude adjustment and propulsion of the unmanned aerial vehicle according to the action instructions and real-time perception of current flight state information including attitude angle, flight speed, and flight height data for closed-loop correction and experience sample construction; a double dynamic memory (DDM) mechanism for dynamically storing and scheduling flight experience samples to improve sample utilization efficiency and convergence stability of strategy training, comprising a data classification module and a data replay module: the data classification module for dividing experience data into three categories of result experience, dangerous experience, and safe experience according to flight results; the data replay module for constructing a double mini-batch shunt structure based on the layer field in the sample to optimize parameters of the high-level DQN network and the low-level LSTM+PPO strategy network.

2. The system of claim 1, wherein, The high-level strategy submodule is configured to receive state vectors at a low frequency and evaluate Q values of each candidate direction in the navigation direction set using a deep Q network (DQN) based reinforcement learning method to select an optimal navigation intention for guiding the low-level strategy to generate fine-grained control actions and realize macro-level path planning.

3. The system of claim 1, wherein, The low-level strategy submodule is configured to receive the navigation intention and state time sequence output by the high-level module at a high frequency and output three-dimensional continuous action control instructions including yaw angular velocity, pitch angular velocity, and thrust control value based on a neural network model combining a long short-term memory (LSTM) network and a proximal policy optimization (PPO) algorithm to realize dynamic adjustment and obstacle avoidance of micro flight behavior of the unmanned aerial vehicle.

4. The system of claim 1, wherein, The double dynamic memory (DDM) mechanism is composed of a data classification module and a data replay module and has two layers of dynamic processing capabilities: the first layer of dynamics is dynamic classification and storage of experience samples; the second layer of dynamics is a double mini-batch structure constructed according to experience types to realize differentiated sampling and hierarchical strategy optimization; to preserve and utilize key decision states to improve stability and strategy generalization ability in the training process.

5. The system of claim 4, wherein, The data classification module automatically classifies experience samples in real time based on flight results, forming a first layer of dynamic processing mechanism, including: Marking trajectories reaching the target or colliding as result experience RE; Marking trajectories with a distance to obstacles less than a preset threshold as dangerous experience DE; Marking the rest in normal state as safe experience SE; And setting different retention probabilities, specifically PRE = 0.95, PDE = 0.10, PSE = 0.01, where PRE, PDE and PSE are the retention probabilities of result experience, dangerous experience and safe experience respectively.

6. The system of claim 4, wherein, The data playback module constitutes a second layer of dynamic processing mechanism, which is used to optimally sample from the three types of experience pools at a set ratio to construct a double-mini-batch structure for policy optimization: High-level mini-batch is used for value function update of DQN network; Low-level mini-batch is used for gradient optimization of LSTM+PPO policy network; The sampling ratio is RE:DE:SE = 60:30:10, which can be dynamically adjusted according to the training stage to achieve the gradual training goal from initial stable control to later policy reinforcement.

Images