A method and system for dynamic path optimization control of a UAV based on reinforcement learning

Abstract

This invention discloses a method and system for dynamic trajectory optimization control of unmanned aerial vehicles (UAVs) based on reinforcement learning, belonging to the field of UAV trajectory optimization technology. The control method includes: data acquisition: real-time acquisition of the target object's current state characteristics and subsequent predicted state characteristics; state construction: constructing a current state vector by combining the relative position characteristics, state characteristics, and predicted state characteristics between the UAV and the target object. This invention constructs a policy network containing mutually exclusive stable actions and diverse actions, and introduces a dynamic integral adjustment algorithm that adjusts the weight coefficients of the reward function in real-time based on the target object's motion state. Utilizing the instantaneous rate of change of the reward integral as a feedback signal, it drives the policy network to actively trigger action switching when the integral acquisition rate decreases, significantly improving the efficiency of accumulated reward integrals and the long-term reward of the tracking task.

Description

Technical Field

[0001] This invention relates to the field of UAV trajectory optimization technology, specifically to a UAV dynamic trajectory optimization control method and system based on reinforcement learning. Background Technology

[0002] Reinforcement learning learns optimal strategies through the interaction between the agent on the drone and the environment, enabling end-to-end trajectory decision-making without relying on a precise environmental model.

[0003] The patent classification number is "CN112131661A" and the name is "A method for UAV autonomously tracking and filming moving targets". The above patent includes (1) constructing a UAV simulator; (2) collecting samples in the UAV simulator; (3) using the collected samples to train the UAV flight control strategy. In the process of training the UAV flight control strategy, a neural network is used to represent the initial UAV flight control strategy model. The UAV is controlled in the UAV simulator using the current initial flight control strategy model. The UAV is sampled in the Markov process provided by the UAV simulator. For the collected samples, the current initial flight control strategy model is optimized using the near-end strategy optimization method until the initial flight control strategy model no longer improves. The UAV flight control strategy model is obtained. The flight control strategy learned by reinforcement learning is often able to cope with various complex situations due to the rich training samples. It has the advantages of flexible response, stable control and low manpower cost.

[0004] The strategy network of the aforementioned patent outputs a single operation command (roll, pitch, yaw, power), which cannot actively plan a trajectory that deviates from the current relative pose during the execution of a stable tracking mission. As a result, the UAV tends to maintain a fixed relative position and shooting angle throughout the tracking process, making it difficult to obtain diverse composition materials. To address this, a UAV dynamic trajectory optimization control method and system based on reinforcement learning has been invented. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for dynamic trajectory optimization control of unmanned aerial vehicles based on reinforcement learning, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a dynamic trajectory optimization control method for unmanned aerial vehicles based on reinforcement learning, the control method comprising:

[0007] Data acquisition: Real-time acquisition of the target object's current state characteristics and subsequent predicted state characteristics;

[0008] State construction: The relative position features, state features, and predicted state features between the UAV and the target object are combined to form the current state vector;

[0009] Action Decision and Policy Network Training: The current state vector is input into a pre-trained policy network, which outputs the UAV's target actions. These target actions include mutually exclusive stable actions and diverse actions. The policy network is trained to maximize the cumulative reward score through a reward function. Calculate the instant reward points and accumulate them to obtain cumulative reward points. Stable actions and diverse actions correspond to the stable action points and diverse action points in the reward function, respectively. The instantaneous rate of change of the instant reward points. As a driving force, the strategy network adjusts the feedback signal of the target action;

[0010] Dynamic Constraints and Weight Adjustment: The dynamic integral adjustment algorithm adjusts the relative weight coefficients of the stability action integral, diversity action integral, and stability action integral in the reward function based on the motion state of the target object in the current state vector. Based on the motion state of the target object, a dynamic state constraint boundary is generated to constrain the relative position features. An ideal constraint subspace is established within the dynamic state constraint boundary. The reward function is then controlled based on the relative position features and the dynamic state constraint boundary. Points for rewards within the system;

[0011] Track correction mechanism: Update the dynamic state constraint boundary based on the change of the current state vector, determine whether the expected target action of the UAV exceeds the dynamic state constraint boundary, and when it is determined to exceed the dynamic state constraint boundary, guide the UAV into the dynamic state constraint boundary through the target state guidance method.

[0012] Furthermore, the reward function include:

[0013] ;

[0014] The total integral at time t is For the first Stability integral of actions at time points, Integral the diversity of actions at time t. To the motion state of the target object Associated dynamic weighting coefficients, ∈(0,1 , For the first Reward points for specific time points The fixed weighting coefficient for the reward item points, For the first The penalty item integral at the time point, The fixed weighting coefficients for the integral of the penalty term;

[0015] The stability action is used to reward the duration for which the drone maintains a fixed shooting angle on the target object, and the diversity action is used to reward the drone's behavior of switching to a new shooting angle with a difference exceeding a preset difference threshold.

[0016] Accumulated reward points are earned through a points system, which includes:

[0017] ;

[0018] The cumulative reward points are calculated from the start of filming until the end of filming at time T.

[0019] The instantaneous rate of change of reward points is obtained based on the reward function, and the trend of the reward function is obtained based on the instantaneous rate of change. When the instantaneous rate of change is detected to be less than the rate of change threshold, the switching between drone stability actions and diversity actions is triggered.

[0020] Furthermore, the instantaneous rate of change include:

[0021] ;

[0022] This indicates that the instant reward value occurs over time. The instantaneous rate of change at that point For time Instant reward value, For time Instant reward value, This is considered the minimum sampling period for the sensors on the drone used to monitor the motion state of the target object.

[0023] Furthermore, To the motion state of the target object The associated dynamic weighting coefficients include:

[0024] Extract the target motion state acceleration variance and rate of change of angular velocity Constructing a motion stability index ;

[0025] Set the first stationarity threshold With the second stationarity threshold ,and ;

[0026] when At that time, if the target is determined to be in a high-dynamic motion phase, set... 1. Generate dynamic state constraint boundaries for maintaining stable drone following;

[0027] when When the target is determined to be in a stable motion phase, set... 0, Generate dynamic state constraint boundaries to support drone-perspective exploration;

[0028] when When the target is determined to be in a transitional motion phase, the system performs the following steps during this phase:

[0029] The first step is to set a fixed time interval. ;

[0030] The second step is to... at each time interval At the beginning, calculate the probability value. ;

[0031] The third step is to randomly generate a random number that is uniformly distributed within the interval [0,1]. ;

[0032] Fourth step, if r≤ Then set ,like Then set ;

[0033] Fifth step, to... The value remains constant until one of the following conditions is met:

[0034] (a) The next time interval start;

[0035] (b) The motion stability index or ;

[0036] The boundary dimensions of dynamic state constraint boundaries also vary. It changes in a stepped manner, with N discrete step positions preset. Each discrete step position corresponds to a boundary dimension of a dynamic state constraint boundary. The boundary dimension increases as the step number increases. The range of values ​​is from arrive The system is divided into N consecutive and non-overlapping sub-intervals. Each sub-interval corresponds to a gear position. The gear positions are assigned to each sub-interval in ascending order of their numerical values, from the 1st to the Nth gear position. N is greater than or equal to 2 and is a positive integer.

[0037] Furthermore, the instantaneous rate of change of instant reward points. As a feedback signal driving the policy network to adjust the target action:

[0038] The preset rate of change threshold is 0;

[0039] Real-time calculation of the instantaneous rate of change of instant reward points △R(t) ;

[0040] when When the value is less than 0, it is determined that the current score acquisition rate has decreased, triggering the policy network to switch between stability actions and diversity actions.

[0041] when When the value is ≥0, the current target action remains unchanged.

[0042] Furthermore, the target state guidance method includes:

[0043] The updated current state vector is input into the policy network, and the policy network outputs the updated dynamic state constraint boundary.

[0044] The base point region is selected based on the nearest principle: based on the current relative position features between the UAV and the target object, and using the subsequent predicted state features of the target object, the dynamic distance between the current position of the UAV and each region within the updated dynamic state constraint boundary under the target's predicted motion trajectory is calculated, and the region with the smallest dynamic distance is selected as the base point region.

[0045] The policy network directly outputs the target action for controlling the UAV based on the updated current state vector, and controls the UAV to fly from the current position to the base point area.

[0046] As the drone flies toward the base point area, the position of the base point area is dynamically adjusted based on the real-time updated predicted state characteristics of the target object until the drone enters the updated dynamic state constraint boundary.

[0047] Furthermore, the ideal constraint subspace inside the dynamic state constraint boundary:

[0048] The entire boundary surface of the dynamic state constraint boundary is uniformly offset inward along its normal direction by a preset distance to form a nested sub-region, thereby obtaining an ideal constraint subspace.

[0049] The distance offset is dynamically adjusted according to the movement speed of the target object: when the movement speed of the target object increases, the distance offset increases accordingly, thereby expanding the ideal constraint subspace;

[0050] When the target object's speed decreases, the distance offset decreases accordingly, thus shrinking the ideal constraint subspace.

[0051] A reinforcement learning-based dynamic trajectory optimization control system for unmanned aerial vehicles (UAVs), the control system comprising:

[0052] Data acquisition module: Real-time acquisition of the target object's current state characteristics and subsequent predicted state characteristics;

[0053] State construction module: Combines the relative position features, state features, and predicted state features between the UAV and the target object to form the current state vector;

[0054] Action Decision and Policy Network Training Module: The current state vector is input into a pre-trained policy network, which outputs the target actions of the UAV. These target actions include mutually exclusive stable actions and diverse actions. The policy network is trained to maximize the cumulative reward score through a reward function. Calculate the instant reward points and accumulate the cumulative reward points. The stable action and the diverse action correspond to the stable action integral and the diverse action integral in the reward function, respectively. The instantaneous rate of change of the instant reward points. As a driving force, the strategy network adjusts the feedback signal of the target action;

[0055] Dynamic Constraints and Weight Adjustment Module: The dynamic integral adjustment algorithm adjusts the relative weight coefficients of the stability action integral, diversity action integral, and stability action integral in the reward function based on the target object's motion state in the current state vector. Based on the motion state of the target object, a dynamic state constraint boundary is generated to constrain the relative position features. An ideal constraint subspace is established within the dynamic state constraint boundary. The reward function is then controlled based on the relative position features and the dynamic state constraint boundary. Points for rewards within the system;

[0056] The trajectory correction mechanism module updates the dynamic state constraint boundary based on the change of the current state vector, determines whether the expected target action of the UAV exceeds the dynamic state constraint boundary, and guides the UAV into the dynamic state constraint boundary through the target state guidance method when it is determined that it exceeds the dynamic state constraint boundary.

[0057] Compared with the prior art, the beneficial effects of the present invention are:

[0058] This reinforcement learning-based UAV dynamic trajectory optimization control method and system constructs a policy network containing mutually exclusive stable actions and diverse actions. It introduces a dynamic integral adjustment algorithm that adjusts the weight coefficients of the reward function in real time based on the target object's motion state. This enables intelligent adaptive switching between "steady-state following" and "dynamic composition" modes for the UAV. When the target is in high-dynamic motion, the system prioritizes stability gains, forcing the UAV to maintain a fixed shooting angle to ensure continuous and uninterrupted footage. When the target is in stable motion, the system prioritizes diversity gains, encouraging the UAV to actively explore novel compositions and acquire rich materials. Utilizing the instantaneous rate of change of reward points as a feedback signal, the system drives the policy network to actively trigger action switching when the point acquisition rate decreases. This effectively avoids the UAV falling into the local predicament of a single-gain mode, significantly improving the efficiency of accumulated reward points and the long-term returns of the tracking task.

[0059] Simultaneously, a dynamic state constraint boundary is constructed with the target object as the base point and adjusted in real time according to its motion state. A nested ideal constraint subspace is established within it. When the target motion state changes and causes the constraint boundary to be updated, the system can determine whether the expected target action of the UAV exceeds the new boundary. Once it is determined that it exceeds the boundary, the target state guidance method is immediately executed. The updated dynamic state constraint boundary and base point area are output through the policy network, guiding the UAV to fly directly into the area from the current position without relying on complex trajectory replanning. This mechanism effectively solves the problems of delayed response and suboptimal path correction in existing methods after exceeding the tracking range, ensuring that the UAV is always in a safe and effective area with acceptable shooting quality, realizing dynamic trajectory correction and safe and continuous optimized following.

[0060] By introducing a motion stability index (combining acceleration variance and angular velocity rate of change) to quantify and classify the target motion state, and setting corresponding step-changing dynamic state constraint boundary dimensions and probabilistic weight coefficient switching strategies, this system can provide a narrow constraint range that closely follows the target to ensure tracking reliability when the target motion uncertainty is high, and provide a loose constraint range to support viewpoint exploration when the target motion predictability is high. The ideal constraint subspace that dynamically adjusts the distance offset with the target speed provides an adaptive buffer margin for the UAV, reducing the risk of boundary violation penalties. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the control method of the present invention;

[0062] Figure 2 This is a schematic diagram of the dynamic weighting coefficients of the present invention;

[0063] Figure 3 This is a schematic diagram illustrating how the present invention obtains the ideal constrained subspace;

[0064] Figure 4 This is a schematic diagram of the reward function of the present invention;

[0065] Figure 5 This is a schematic diagram of the dynamic state constraint boundary of the present invention;

[0066] Figure 6 This is a schematic diagram of the target guidance method of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] like Figure 1 - Figure 6 As shown, this invention provides a technical solution: a dynamic trajectory optimization control method for unmanned aerial vehicles based on reinforcement learning, the control method comprising:

[0069] Data acquisition: Real-time acquisition of the target object's current state characteristics and subsequent predicted state characteristics;

[0070] State construction: The relative position features, state features, and predicted state features between the UAV and the target object are combined to form the current state vector;

[0071] Action Decision and Policy Network Training: The current state vector is input into a pre-trained policy network. The policy network outputs the target actions of the UAV, which include mutually exclusive stable actions and diverse actions. The policy network is trained to maximize the cumulative reward points through a reward function. Calculate the instant reward points and accumulate them to obtain cumulative reward points. Stable actions and diverse actions correspond to the stable action points and diverse action points in the reward function, respectively. The instantaneous rate of change of the instant reward points. As a driving force, the strategy network adjusts the feedback signal of the target action;

[0072] Dynamic Constraints and Weight Adjustment: The dynamic integral adjustment algorithm adjusts the relative weight coefficients of the stability action integral, diversity action integral, and stability action integral in the reward function based on the motion state of the target object in the current state vector. Based on the motion state of the target object, a dynamic state constraint boundary is generated to constrain the relative position features. An ideal constraint subspace is established within the dynamic state constraint boundary. The reward function is then controlled based on the relative position features and the dynamic state constraint boundary. Points for rewards within the system;

[0073] Track correction mechanism: Update the dynamic state constraint boundary based on the change of the current state vector, determine whether the expected target action of the UAV exceeds the dynamic state constraint boundary, and when it is determined to exceed the dynamic state constraint boundary, guide the UAV into the dynamic state constraint boundary through the target state guidance method.

[0074] reward function include:

[0075] ;

[0076] The total integral at time t is For the first Stability integral of actions at time points, Integral the diversity of actions at time t. To the motion state of the target object Associated dynamic weighting coefficients, ∈(0,1 , For the first Reward points for specific time points The fixed weighting coefficient for the reward item points, For the first The penalty item integral at the time point, The fixed weighting coefficients for the integral of the penalty term;

[0077] Stability actions are used to reward the duration for which the drone maintains a fixed shooting angle on the target object, while diversity actions are used to reward the drone's behavior of switching to a new shooting angle with a difference of more than a preset difference threshold from the historical shooting angle.

[0078] Accumulated reward points are earned through a points system, which includes:

[0079] ;

[0080] The cumulative bonus points are calculated from the start of filming until the end of filming, T.

[0081] The instantaneous rate of change of reward points is obtained based on the reward function, and the trend of the reward function is obtained based on the instantaneous rate of change. When the instantaneous rate of change is detected to be less than the rate of change threshold, the switching between drone stability actions and diversity actions is triggered.

[0082] instantaneous rate of change include:

[0083] ;

[0084] Indicates the instant reward value over time. The instantaneous rate of change at that point For time Instant reward value, For time Instant reward value, This is considered the minimum sampling period for the sensors on the drone used to monitor the motion state of the target object.

[0085] To the motion state of the target object The associated dynamic weighting coefficients include:

[0086] Extracting the target motion state acceleration variance and rate of change of angular velocity Constructing a motion stability index ;

[0087] Set the first stationarity threshold With the second stationarity threshold ,and ;

[0088] when At that time, if the target is determined to be in a high-dynamic motion phase, set... 1. Generate dynamic state constraint boundaries for maintaining stable drone following;

[0089] when When the target is determined to be in a stable motion phase, set... 0, Generate dynamic state constraint boundaries to support drone-perspective exploration;

[0090] when When the target is determined to be in a transitional motion phase, the system executes the following steps during this phase:

[0091] The first step is to set a fixed time interval. ;

[0092] The second step is to... at each time interval At the beginning, calculate the probability value. ;

[0093] The third step is to randomly generate a random number that is uniformly distributed within the interval [0,1]. ;

[0094] Fourth step, if r≤ Then set ,like Then set ;

[0095] Fifth step, The value remains constant until one of the following conditions is met:

[0096] (a) The next time interval start;

[0097] (b) Motion stability index or ;

[0098] The boundary dimensions of dynamic state constraint boundaries also vary. It changes in a stepped manner, with N discrete step positions preset. Each discrete step position corresponds to a boundary dimension of a dynamic state constraint boundary. The boundary dimension increases as the step number increases. The range of values ​​is from arrive The system is divided into N consecutive and non-overlapping sub-intervals. Each sub-interval corresponds to a gear position. The gear positions are assigned to each sub-interval in ascending order of their numerical values, from the 1st to the Nth gear position. N is greater than or equal to 2 and is a positive integer.

[0099] Instantaneous change rate of instant reward points As a feedback signal driving the policy network to adjust the target action:

[0100] The preset rate of change threshold is 0;

[0101] Real-time calculation of the instantaneous rate of change of instant reward points △R(t) ;

[0102] when When the value is less than 0, it is determined that the current score acquisition rate has decreased, triggering the policy network to switch between stability actions and diversity actions.

[0103] when When the value is ≥0, the current target action remains unchanged.

[0104] Target state guidance methods include:

[0105] The updated current state vector is input into the policy network, and the policy network outputs the updated dynamic state constraint boundary.

[0106] The base point region is selected based on the principle of proximity: based on the current relative position features between the UAV and the target object, and using the subsequent predicted state features of the target object, the dynamic distance between the current position of the UAV and each region within the updated dynamic state constraint boundary under the predicted motion trajectory of the target is calculated, and the region with the smallest dynamic distance is selected as the base point region.

[0107] The policy network directly outputs the target action to control the drone based on the updated current state vector, controlling the drone to fly from its current position to the base point area;

[0108] As the drone flies toward the base point area, the position of the base point area is dynamically adjusted based on the real-time updated predicted state characteristics of the target object until the drone enters the updated dynamic state constraint boundary.

[0109] Ideal constraint subspace inside the dynamic state constraint boundary:

[0110] The entire boundary surface of the dynamic state constraint boundary is uniformly offset inward along its normal direction by a preset distance to form a nested sub-region, thus obtaining the ideal constraint subspace.

[0111] The distance offset is dynamically adjusted according to the target object's movement speed: when the target object's movement speed increases, the distance offset increases accordingly, thus expanding the ideal constraint subspace;

[0112] When the target object's speed decreases, the distance offset decreases accordingly, thus shrinking the ideal constraint subspace.

[0113] A reinforcement learning-based dynamic trajectory optimization control system for unmanned aerial vehicles (UAVs), the control system comprising:

[0114] Data acquisition module: Real-time acquisition of the target object's current state characteristics and subsequent predicted state characteristics;

[0115] State construction module: Combines the relative position features, state features, and predicted state features between the UAV and the target object to form the current state vector;

[0116] Action Decision and Policy Network Training Module: The current state vector is input into a pre-trained policy network. The policy network outputs the target actions of the UAV, including mutually exclusive stable actions and diverse actions. The policy network is trained to maximize the cumulative reward points through a reward function. Calculate the instant reward points and accumulate the cumulative reward points. Stable actions and diverse actions correspond to the stable action points and diverse action points in the reward function, respectively. Calculate the instantaneous rate of change of the instant reward points. As a driving force, the strategy network adjusts the feedback signal of the target action;

[0117] Dynamic Constraints and Weight Adjustment Module: The dynamic integral adjustment algorithm adjusts the relative weight coefficients of the stability action integral, diversity action integral, and stability action integral in the reward function based on the target object's motion state in the current state vector. Based on the motion state of the target object, a dynamic state constraint boundary is generated to constrain the relative position features. An ideal constraint subspace is established within the dynamic state constraint boundary. The reward function is then controlled based on the relative position features and the dynamic state constraint boundary. Points for rewards within the system;

[0118] The trajectory correction mechanism module updates the dynamic state constraint boundary based on the change of the current state vector, determines whether the expected target action of the UAV exceeds the dynamic state constraint boundary, and guides the UAV into the dynamic state constraint boundary through the target state guidance method when it is determined to exceed the dynamic state constraint boundary.

[0119] like Figure 1 As shown, the UAV collects the motion state of the target object in real time and predicts its future motion state within a preset time period. The current flight pose of the UAV, the current shooting angle, the current and future motion state of the target object, and the relative position between them constitute the environmental state. The environmental state is input into a pre-constructed policy network trained with the goal of maximizing the cumulative reward score. The network outputs and controls the UAV to perform target actions. A dynamic state constraint boundary is generated based on the motion stability index and relative weight coefficients, with the target object as the base region, to restrict the UAV's position. A corresponding ideal constraint subspace is established within this boundary. The UAV is controlled to stay within the ideal constraint subspace as much as possible. The cumulative reward score and its instantaneous rate of change are calculated using a reward function. Based on the environmental state of the target object, a dynamic integral adjustment algorithm is used to adjust the stability action integral and the diversity action integral, as well as their relative weight coefficients in the reward function, in real time. This allows the UAV to switch between stable actions and diversity actions within the dynamic state constraint boundary. When based on the motion state... After updating the dynamic state constraint boundary, if it is determined that the drone's position will exceed the dynamic state constraint boundary, the target state guidance method is executed. The target action is output to control the drone to move directly to the base point area, thereby completing the dynamic correction and optimized following of the trajectory. A policy network containing mutually exclusive stable actions and diverse actions is constructed, and a dynamic integral adjustment algorithm based on the target object's motion state to adjust the weight coefficient of the reward function in real time is introduced. This enables the drone to intelligently and adaptively switch between the "steady-state following" and "dynamic composition" modes. When the target is in high dynamic motion, the system focuses on stability gains, forcing the drone to maintain a fixed shooting angle to ensure continuous and unlost footage. When the target is in stable motion, the system focuses on diversity gains, encouraging the drone to actively explore novel compositions and acquire rich materials. The instantaneous change rate of the reward points is used as a feedback signal to drive the policy network to actively trigger action switching when the points acquisition rate decreases. This effectively avoids the drone getting stuck in a local dilemma of a single-gain mode, significantly improving the efficiency of accumulated reward points and the long-term return of the tracking task.

[0120] The UAV can obtain motion state parameters such as the three-dimensional position, velocity, acceleration, azimuth, pitch angle and angular velocity of the target object relative to the UAV in real time through visual sensors. These parameters are the relative position features and the current state features. Through the internal calculation module, and with the target motion sequence within a historical time window (such as the past 10 frames) as input, the motion state at several future moments is directly output to obtain the subsequent predicted state features. The current flight pose of the UAV, the current shooting angle, the current and future motion state of the target object and the relative position between the two are combined to form the current state vector, which is input into a pre-constructed policy network trained with the goal of maximizing the cumulative reward points. The policy network outputs and controls the UAV to execute the target action.

[0121] When a drone automatically follows and films without needing obstacle avoidance, its flight path is primarily influenced by two factors: First, the relative position-maintaining trajectory, generated based on the target's motion state. Its goal is to maintain a fixed relative pose between the drone and the target, thus obtaining stable and continuous footage. Second, the independently planned trajectory for the drone to actively change its shooting angle towards the target. The introduction of second trajectory aims to overcome the limitations of a fixed perspective by actively deviating from the current relative pose to fly to a new spatial location significantly different from historical shooting angles, thereby acquiring diverse compositional material. When the system determines that the current target's motion is stable and suitable for perspective exploration (i.e., dynamic weighting coefficient), the drone's trajectory is adjusted accordingly. (Lower), the policy network generates a set of target actions that deviate from the current stable following pose under the drive of the diversity actions of the reward function. The target actions make the UAV move along a predetermined path, continuously changing the azimuth and pitch angles relative to the target object during the movement until it reaches the target view position. By implicitly fusing the No. 1 trajectory and the No. 2 trajectory in the action output of the policy network - that is, the target action output by the policy network simultaneously includes the component of maintaining relative position (corresponding to stability gain) and the component of active trajectory change (corresponding to diversity gain), this application is an optimization for the No. 2 trajectory.

[0122] During the tracking and filming of the target object, the UAV acquires the real-time motion state of the target object and executes a dual-modal position control strategy: In steady-state following mode, it adjusts its own posture in the opposite direction based on the target's motion state to maintain a fixed relative position with the target object, i.e., a stability action, thereby obtaining stability gain points; in dynamic composition mode, it actively plans a change-track trajectory based on the target's motion state, breaking the original relative position relationship to perform angle switching filming, i.e., a diversity action, thereby obtaining diversity gain points. Note that even if the target object moves but the UAV's position is fixed, diversity gain points will still be obtained. The UAV only obtains stability gain points in steady-state following mode, while the diversity gain points are zero in this mode.

[0123] Stability action integral When the drone's shooting angle (including horizontal azimuth and pitch) relative to the target object remains constant (i.e. the angle change is less than a preset stability threshold, such as ±2°), +1 point is awarded at each time point.

[0124] Diverse motion integrals When a drone actively switches its shooting angle, and the difference between the new angle and the angle from which it last gained diversity benefits exceeds a preset switching threshold (e.g., azimuth change > 30° or pitch change > 15°), it receives +1 point. At the same time, the system records the angle from which diversity benefits were last gained. If the drone switches back to an angle from which diversity benefits were previously gained within a short period of time (e.g., within 1 minute), it will not receive points for switching back and forth, and may even receive a small negative point (e.g., -0.5 points) to prevent the drone from repeatedly switching back and forth between two fixed angles to rack up points. Only when switching to a completely new angle that is significantly different from all previous angles from which diversity benefits were gained will a positive point be awarded.

[0125] The bonus points are as follows: 1 point is awarded when the drone is within the dynamic state constraint boundary but not within the ideal constraint subspace; 2 points are awarded when the drone is within the ideal constraint subspace.

[0126] The penalty points include -2 points for leaving the dynamic state constraint boundary, and corresponding deductions based on the distance between the drone and the dynamic state constraint boundary, and -100 points for colliding with an obstacle.

[0127] Target object motion state Corresponding dynamic weight coefficients =0.7, stability action integral =1.0 (The drone maintains a fixed shooting angle, which is relative to the target object), diversity motion integral =0 (current view not switched), distance-preserving term integral =2.0 (the drone is within the ideally constrained subspace), penalty term integral (Within the boundary), fixed weighting coefficient =0.5, =0.8;

[0128] =0.7×1.0+(1−0.7)×0+0.5×2.0−0.8×0=0.7+0+1.0=1.7, the score obtained at this moment is 1.7. If the drone and the target object can maintain the current state for 0.5 seconds, =1.7 × 0.5 = 0.85, the cumulative integral in that 0.5 seconds is 0.85.

[0129] like Figure 4 As shown, assuming that from the start of the tracking and filming at k=0 to the current time T=5 seconds (the minimum sampling period of the sensor on the drone used to monitor the motion state of the target object is considered to be 0.1 seconds, and the corresponding processing period is also 0.1 seconds, for a total of 50 time points), at each time point... The following are time points: 1 to 10. Stable at 1.600; time points 11-20. It stabilized at 1.700; timeframe: 21-30. Stable at 1.500; time point 31-40. Stable at 1.800; time point 41-50, Stabilizing at 1.600, with straight lines connecting adjacent time points, the cumulative integral from time points 1 to 10 is 1.600 × 1 = 0.165. The integrals at time points 10 and 11 are considered to be directly connected, so the integral from time points 10 to 11 is (1.600 + 1.700) × 0.5 × 0.1 = 0.165. The same logic applies to subsequent times.

[0130] =1.600+0.165+1.530+0.160+1.350+0.165+1.620+0.170+1.440=8.200:

[0131] Calculate the instantaneous rate of change at time point 21 , =(1.500-1.700) / 0.1=-2.0, -2.0 is less than 0, so the switching between stable actions and diverse actions is realized.

[0132] In the policy network construction and training, reinforcement learning is the foundation, with maximizing the cumulative reward integral being a key aspect of reinforcement learning. Training takes place in a simulation environment where the target object's trajectory can be simulated using real-world data. The simulation environment provides real-time data on the relative pose, velocity, acceleration, camera viewpoint, and boundary parameters of the dynamic state constraints between the UAV and the target. This includes the UAV's 3D position relative to the target, relative velocity, relative acceleration, current camera viewpoint (azimuth and pitch), the target object's trajectory over several past frames, and the boundary dimensions of the current dynamic state constraints. Reinforcement learning is primarily manifested in modeling the UAV dynamic trajectory optimization problem. This is achieved by constructing an environmental state vector containing the relative position, motion state, and predicted state of the UAV and the target object as input, and designing a policy network with the objective of maximizing the cumulative reward integral. The network output... The system identifies mutually exclusive stable actions (maintaining a fixed shooting angle) and diverse actions (actively switching to new angles). Simultaneously, a carefully designed reward function (including integrals for stable actions, diverse actions, distance maintenance, and penalties) calculates immediate rewards. The instantaneous rate of change of the reward integral is introduced as a feedback signal driving the policy network to switch between the two mutually exclusive actions. A dynamic integral adjustment algorithm adaptively adjusts the relative weight coefficients of each action integral in the reward function based on the target object's motion stability index. This allows the UAV to autonomously learn the optimal trajectory decision strategy through continuous interaction with the environment: emphasizing stable following to obtain continuous rewards when the target is in high-dynamic motion, and actively exploring novel angles to obtain diverse rewards when the target is in stable motion. The system guides trajectory correction through dynamic state constraint boundaries and ideal constraint subspaces, ultimately achieving autonomous optimization control of the UAV following the trajectory.

[0133] In each training round, the initial position of the UAV is randomly set (usually within an ideal constraint subspace inside the dynamic state constraint boundary), the motion state of the target object and the predicted future motion state are initialized, the environmental state is input into the policy network, the motion of the target object is simulated in the simulation environment, the UAV performs actions, the simulation environment updates the poses of the UAV and the target object, and the instantaneous reward is calculated. When the drone performs actions that help maintain stable tracking (i.e., obtain a high stability action score), Or successfully switch to a novel shooting perspective (i.e., obtain a higher diversity action score). ), and always within the dynamic state constraint boundary, If the value is positive, the accumulated reward points will increase accordingly. After completing the task, the calculation will be performed. Accumulated reward points are awarded, and the instantaneous rate of change of reward points is introduced.

[0134] The drone accumulates reward points during the follow-up filming mission. It increases monotonically over time (primarily with positive rewards), and It is generally always positive, and the reward function also includes the stability action integral. and diversity of motion integrals Furthermore, there is a competitive relationship between the two (it is difficult to obtain diverse benefits when maintaining stable tracking, and vice versa). Drones are prone to getting stuck in a local dilemma: that is, staying in one benefit mode for a long time, causing the point acquisition rate to gradually decrease. However, since there are still point benefits, the drone will not switch between stable and diverse actions. If the scene itself changes slowly (such as the target object moving in a straight line at a constant speed), continuing to maintain stable tracking can still obtain a small number of points, but the point acquisition rate (i.e., the average reward per unit time) is not high. If it can switch to a diverse benefit mode, it can obtain +1 point through a new perspective change (equivalent to a short-term surge in the point acquisition rate), and can obtain a higher cumulative point in the same time. Instantaneous change rate is introduced to quantify "whether the current point acquisition rate has decreased", and this is used as the basis for triggering the switch of benefit mode. The change rate threshold is set to zero. When it is greater than 0, it means The value is increasing, no switching is needed, when... When it equals 0, it means The value is in a stable state when When the value is less than 0, the system switches between stable and diverse actions to ensure the acquisition of points. The reward points define the drone's immediate gains at every moment, serving as the "behavioral guidelines" for the policy network's learning. The instantaneous change rate monitors the growth rate of points in real time. Once it detects a continuous decline in the growth rate of points, it proactively triggers the switch between stable and diverse actions, essentially issuing a prompt: "The efficiency of the current strategy's gains is decreasing; please explore new modes." This prompt drives the policy network to break out of local optima (e.g., long-term stable tracking can consistently earn points but lacks explosive gains), thereby trying new actions to obtain higher long-term cumulative rewards. Therefore, the logic of the integration of the two is: the reward points provide fine-grained value assessment, while the instantaneous change rate provides macro-level trend feedback. Together, they guide the drone to intelligently switch between "stability" and "exploration," avoiding falling into a single gain mode.

[0135] like Figure 3 and Figure 5As shown, the dynamic state constraint boundary, which defines the relative positional characteristics of the target object, is used to constrain the mobile space of the UAV since the target object is uncontrollable. This ensures that the UAV remains within a safe area where the shooting quality of the target object meets the standards when performing follow-up shooting tasks. The establishment of this range is achieved by engineers or users who pre-set key parameters according to actual shooting needs and adjust them in real time according to the movement state of the target object and the shooting intention. The parameters include the shortest and longest allowable distance between the UAV and the target object, the allowable relative height range (e.g., how many meters above or below the target), and the allowable horizontal azimuth range (i.e., the angle range in which the UAV is allowed to appear in front of, to the side of, or behind the target object). Actual shooting needs may vary depending on the usage scenario, such as mountain biking, skiing, and rock climbing, and the above parameters will also be adjusted accordingly.

[0136] Within the established dynamic state constraint boundary, a nested sub-region is defined as the ideal constraint subspace. The ideal constraint subspace is constructed by equidistant offset and using a velocity ratio. The entire boundary (or all boundary surfaces) of the dynamic state constraint boundary is uniformly offset inward along its normal direction by a preset and variable distance offset, so that the UAV is preferentially guided to operate within this region to obtain sufficient out-of-bounds buffer margin. The velocity ratio, i.e., the variable distance offset, is dynamically adjusted according to the movement speed of the target object, thereby adjusting the ideal constraint subspace. The faster the speed, the larger the ideal constraint subspace (providing a wider buffer fluctuation area for the UAV), and the slower the speed, the smaller the ideal constraint subspace (more precise control and reduced redundant maneuvers).

[0137] like Figure 6 As shown, there is a significant time delay from when the UAV senses the target's state, calculates control commands, to when the actuators (motors, control surfaces) produce actual displacement. If the UAV only follows the target based on its current state, it will always be "half a beat behind" when the target suddenly accelerates, decelerates, or turns, leading to relative position deviation or even losing track of the target. By predicting the target's motion state at several future moments, the UAV can plan its trajectory in advance, achieving "advance compensation" to maintain stable following. When the target's changes are small, the predicted dynamic state constraint boundary and the actual dynamic state constraint boundary do not change much and no adjustment is needed. However, when the UAV's position exceeds the predicted dynamic state constraint boundary, situations include significant changes in the target's speed and direction, the UAV exceeding the dynamic state constraint boundary due to obstacle avoidance maneuvers, and changes in the dynamic state constraint boundary caused by switching between multiple targets. Once it is determined that the UAV's position is no longer within the current dynamic state constraint boundary, the policy network outputs target actions to guide the UAV to fly to a new base point area.

[0138] When performing obstacle avoidance maneuvers, the drone still adheres to the constraints of the dynamic state constraint boundary and prioritizes planning obstacle avoidance paths within this range. When the sensor detects an obstacle, the drone first attempts to adjust its heading, speed, or altitude within the current dynamic state constraint boundary to bypass or avoid the obstacle, while ensuring its position remains within the allowable range. Only when there are no collision-free feasible paths within the dynamic state constraint boundary is the drone allowed to temporarily exceed this range to perform emergency obstacle avoidance. During this period, the system marks it as a "temporary obstacle avoidance boundary violation" state. After obstacle avoidance is completed, the drone immediately triggers the target state guidance method and automatically returns to the target base point area within the current dynamic state constraint boundary. For example, when multiple people are rock climbing, each climber (target object) has their own independent position, speed, and direction of movement. When the drone follows the target from the first climber to the second climber, due to the second climber's spatial position, climbing speed, and body position... The orientation differs significantly from the first climber. The dynamic state constraint boundary originally generated based on the first climber is no longer applicable to the new target. The system immediately regenerates a dynamic state constraint boundary based on the current motion state of the second climber, using the new target as the benchmark. This constructs a dynamic state constraint boundary with the target object as the base point and adjusts in real time according to its motion state. Nested ideal constraint subspaces are established within this boundary. When the target's motion state changes, causing the constraint boundary to be updated, the system can determine whether the UAV's expected target action exceeds the new boundary. If it is determined to exceed the boundary, the system immediately executes the target state guidance method, outputting the updated dynamic state constraint boundary and base point region, guiding the UAV directly from its current position into this region without relying on complex trajectory replanning. This mechanism effectively solves the problems of delayed response and suboptimal path correction in existing methods when the target exceeds the tracking range, ensuring that the UAV is always within a safe and high-quality effective area, achieving dynamic trajectory correction and safe, continuous optimized following.

[0139] The preset time period will be adjusted according to the shooting intention and the speed of the target. For example, when skiing, the speed is relatively fast, so the preset time period can be set to 1 second, while when rock climbing, the speed is slow, the preset time period can be set to 0.1 seconds. For high-speed sports scenarios like skiing, the displacement of the target object per unit time is large. If the preset time period is too short (such as 0.1 seconds), the prediction range will not be sufficient to cover the actual movement trend, which may cause the drone to react late and lose track of the target. Appropriately extending the preset time period to 1 second can enable the model to predict the long-term movement trajectory of the target in advance, thereby planning a smoother and more forward-looking trajectory. For low-speed sports scenarios like rock climbing, the target displacement is slow and the movement is highly random. If the preset time period is too long (such as 1 second), the prediction results will contain a lot of uncertainty noise, which will reduce the accuracy of the decision.

[0140] like Figure 2 As shown, the definition Variance of acceleration of the target object With the rate of change of angular velocity The weighted composite index, namely: Within a time window (e.g., 0.5 seconds), the acceleration vector of the target object is continuously acquired. Calculate the magnitude of acceleration or the weighted sum of the variances of its components. The larger the value, the more drastic the acceleration and deceleration of the target, and the more unstable its motion. Real-time monitoring of the rate of change of the target object's angular velocity vector, i.e., angular acceleration, is used to obtain the time derivative of the magnitude of the angular velocity. Calculate the rate of change of angular velocity. The larger the value, the more rapidly the target rotates or changes direction, and the higher the uncertainty of its motion. and | Divide each by its respective maximum expected value Normalize, then multiply by a preset weighting coefficient. , (For example =0.6, =0.4), thus obtaining the stationarity index between 0 and 1. The closer to 1, the smoother the motion; the closer to 0, the more intense and dynamic the motion.

[0141] Set the first stationarity threshold =0.3 and the second stationarity threshold =0.7, when When the value is ≤0.3, the target is determined to be in a high-dynamic motion phase, and the setting is... 1. The reward function focuses entirely on stable actions, and the dynamic state constraint boundary is set to a small range closely following the target, forcing the drone to stay near the optimal following position. When the value is ≥0.7, the target is determined to be in a steady motion phase, and the setting is... 0. The reward function focuses entirely on diverse actions, and the dynamic state constraint boundary is set to a large range, allowing the drone to move freely within the boundary to explore new perspectives.

[0142] When 0.3 < When the value is less than 0.7, the target is determined to be in the transitional motion phase, and the system performs the following steps: Set time interval. =2 seconds; at the beginning of each time interval, calculate A random number r is generated by computer and is uniformly distributed in the interval [0,1]; if r ≤ Then set Otherwise set This setting The value remains fixed until the next time interval begins or the motion stability index crosses the transition range.

[0143] If the target object's acceleration variance =0.25, rate of change of angular velocity =0.3 Substitute into the formula:

[0144] ;

[0145] Calculated When 0.787 ≥ 0.7, the target is determined to be in a steady motion phase, and the setting is... 0;

[0146] If the target object's acceleration variance =1.00, rate of change of angular velocity =0.8 Substitute into the formula:

[0147] ;

[0148] Calculated It falls between 0.3 and 0.7, taking the time interval. =0.5 seconds,

[0149] The system randomly generates r=0.3 (assumed). Since r≤p, we set... =1.

[0150] When the target is in a highly dynamic motion phase ( The target undergoes rapid changes in direction, drastic speed variations, and high motion uncertainty. To ensure image stability and tracking reliability, the reward function should prioritize stability gains; therefore, it should be set... 1 (completely ignoring diversity benefits), while the dynamic state constraint boundary is set to a small range to maintain stable following, that is, a narrow area close to the target object, forcing the drone to stay near the optimal following position and preventing loss of following or image shaking due to large-scale maneuvers.

[0151] When the target is in a steady motion phase ( The target is moving at a constant speed in a straight line or is stationary, and its motion is highly predictable. In this case, the drone can be encouraged to actively explore novel perspectives to enrich the shooting content. The reward function focuses on diversity benefits and is set accordingly. 0. The dynamic state constraint boundary is set to a large range to support viewpoint exploration, allowing the UAV to move within the dynamic state constraint boundary during dynamic operation.

[0152] When the target is in the transitional movement phase ( The target's motion stability is moderate. The reward function changes continuously according to a linear formula, allowing for a smooth transition between stability and diversity. The boundary dimensions of the dynamic state constraint boundary change in a stepped manner, setting multiple discrete step levels (e.g., "small," "medium," and "large"). Each level corresponds to a fixed set of distance and angle boundaries. Only when... Exceeding the preset threshold (e.g.) The system can only switch to the next gear when the target motion state is equal to 0.3. By introducing a motion stability index (combining acceleration variance and angular velocity change rate), the system quantifies and classifies the target motion state. Correspondingly, it sets a step-by-step dynamic state constraint boundary size and a probabilistic weight coefficient switching strategy. This system can provide a narrow constraint range that closely follows the target when the target motion uncertainty is high to ensure tracking reliability, and provide a loose constraint range that supports viewpoint exploration when the target motion predictability is high. The ideal constraint subspace that dynamically adjusts the distance offset with the target speed provides an adaptive buffer margin for the UAV, reducing the risk of exceeding the boundary penalty.

[0153] When a drone performs a shooting mission, the system generates a dynamic and real-time adjustable dynamic state constraint boundary based on the target object. This dynamic state constraint boundary is used to constrain the movement of the drone, ensuring that it always moves within this range. An ideal constraint subspace is also defined inside this dynamic state constraint boundary. By controlling the drone to stay within the ideal constraint subspace as much as possible, even if the trajectory fluctuates or the environmental state of the target object changes, the drone can maintain a certain margin within the dynamic state constraint boundary, thereby avoiding penalties for exceeding the boundary.

[0154] like Figure 6 As shown, if it is determined that the expected trajectory of the UAV will exceed the updated dynamic state constraint boundary, the target state guidance method is executed: the control UAV is transferred to the updated dynamic state constraint boundary, the new environmental state is input into the policy network again to obtain the updated dynamic state constraint boundary and its base point region, and then the target action is output to control the UAV to move directly to the base point region, thereby completing the dynamic correction and optimized following of the trajectory.

[0155] The current state vector is fed into a pre-constructed policy network trained with the goal of maximizing the cumulative reward points. The policy network outputs new dynamic state constraint boundaries and corresponding ideal constraint subspaces in real time according to the environmental state. Based on the changes in the dynamic state constraint boundaries, the UAV is restricted so that it can always stay within the dynamic state constraint boundaries. When calculating the distance between the current position of the UAV and each candidate region within the dynamic state constraint boundaries, not only the spatial coordinate difference at the current moment is considered, but also the predicted motion trajectory of the target in the future time period is included in the calculation based on the predicted state characteristics of the target object. Factors such as the intersection point and relative speed between the predicted target motion trajectory and the candidate region are included to finally obtain a dynamic distance value that changes over time. The candidate region with the smallest dynamic distance is selected as the base region according to the nearest principle.

[0156] In practical applications, dynamic state constraint boundaries not only change with the distribution of environmental obstacles, but also adjust accordingly due to changes in the target direction. Even if the target object itself does not undergo spatial displacement, changes in its direction will reshape the form of the allowable range. For example, the direction of the bicycle's front end. Since the bicycle only moves along the front direction, its dynamic state constraint boundary will change with the front direction. This dynamic state constraint boundary is not a complete circular area constructed with the target object as the base point, but takes into account the target direction and specifically sets a gap in the target direction (i.e., the allowable range presents a specific structure that is open or restricted in this direction) to adapt to non-omnidirectional safety constraints and maneuver requirements.

[0157] This invention system can be widely applied in sports competitions and outdoor sports tracking (such as skiing and downhill mountain biking). In these scenarios, drones need to achieve stable tracking and intelligent switching between diverse perspective shooting for dynamic targets that are moving at high speed or whose attitude changes suddenly, without the need for pilot control. This system dynamically adjusts the reward weight and constraint boundary through reinforcement learning, which can ensure that the image is continuous and not lost when the target is moving violently, automatically explore novel compositions when the target is stable, and ensure flight safety by using dynamic state constraint boundary and ideal subspace. It can also quickly correct the trajectory when the target suddenly changes direction or switches, thereby significantly improving the quality and adaptability of the drone's autonomous tracking.

[0158] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended embodiments and their equivalents.

Claims

1. A method for dynamic trajectory optimization control of unmanned aerial vehicles based on reinforcement learning, characterized in that: The control method includes: Data acquisition: Real-time acquisition of the target object's current state characteristics and subsequent predicted state characteristics; State construction: The relative position features, state features, and predicted state features between the UAV and the target object are combined to form the current state vector; Action Decision and Policy Network Training: The current state vector is input into a pre-trained policy network, which outputs the target actions of the UAV. These target actions include mutually exclusive stable actions and diverse actions. The policy network is trained to maximize the cumulative reward score through a reward function. Calculate the instant reward points and accumulate them to obtain cumulative reward points. Stable actions and diverse actions correspond to the stable action points and diverse action points in the reward function, respectively. The instantaneous rate of change of the instant reward points. As a driving strategy network to adjust the feedback signal of the target action; Dynamic Constraints and Weight Adjustment: The dynamic integral adjustment algorithm adjusts the relative weight coefficients of the stability action integral, diversity action integral, and stability action integral in the reward function based on the motion state of the target object in the current state vector. Based on the motion state of the target object, a dynamic state constraint boundary is generated to constrain the relative position features. An ideal constraint subspace is established within the dynamic state constraint boundary, and the reward function is controlled based on the relative position features and the dynamic state constraint boundary. Points for rewards within the system; Track correction mechanism: Update the dynamic state constraint boundary based on the change of the current state vector, determine whether the expected target action of the UAV exceeds the dynamic state constraint boundary, and when it is determined to exceed the dynamic state constraint boundary, guide the UAV into the dynamic state constraint boundary through the target state guidance method.

2. The method for dynamic trajectory optimization control of unmanned aerial vehicles based on reinforcement learning according to claim 1, characterized in that: The reward function include: ; The total integral at time t is For the first Stability integral of actions at time points, Integral the diversity of actions at time t. To the motion state of the target object Associated dynamic weighting coefficients, ∈(0,1 , For the first Reward points for specific time points The fixed weighting coefficient for the reward item points, For the first The penalty item integral at the time point, The fixed weighting coefficients for the integral of the penalty term; The stability action is used to reward the duration for which the drone maintains a fixed shooting angle on the target object, and the diversity action is used to reward the drone's behavior of switching to a new shooting angle with a difference exceeding a preset difference threshold. Accumulated reward points are earned through a points system, which includes: ; The cumulative reward points are calculated from the start of filming until the end of filming at time T. The instantaneous rate of change of reward points is obtained based on the reward function, and the trend of the reward function is obtained based on the instantaneous rate of change. When the instantaneous rate of change is detected to be less than the rate of change threshold, the switching between drone stability actions and diversity actions is triggered.

3. The method for dynamic trajectory optimization control of unmanned aerial vehicles based on reinforcement learning according to claim 2, characterized in that: The instantaneous rate of change include: ; This indicates that the instant reward value occurs over time. The instantaneous rate of change at that point For time Instant reward value, For time Instant reward value, This is considered the minimum sampling period for the sensors on the drone used to monitor the motion state of the target object.

4. The method for dynamic trajectory optimization control of unmanned aerial vehicles based on reinforcement learning according to claim 2, characterized in that: To the motion state of the target object The associated dynamic weighting coefficients include: Extract the target motion state acceleration variance and rate of change of angular velocity Constructing a motion stability index ; Set the first stationarity threshold With the second stationarity threshold ,and ; when At that time, if the target is determined to be in a high-dynamic motion phase, set...

1. Generate dynamic state constraint boundaries for maintaining stable drone following; when When the target is determined to be in a stable motion phase, set... 0, Generate dynamic state constraint boundaries to support drone-perspective exploration; when When the target is determined to be in a transitional motion phase, the system performs the following steps during this phase: The first step is to set a fixed time interval. ; The second step is to... at each time interval At the beginning, calculate the probability value. ; The third step is to randomly generate a random number that is uniformly distributed within the interval [0,1]. ; Fourth step, if r≤ Then set ,like Then set ; Fifth step, to... The value remains constant until one of the following conditions is met: (a) The next time interval start; (b) The motion stability index or ; The boundary dimensions of dynamic state constraint boundaries also vary. It changes in a stepped manner, with N discrete step positions preset. Each discrete step position corresponds to a boundary dimension of a dynamic state constraint boundary. The boundary dimension increases as the step number increases. The range of values ​​is from arrive The system is divided into N consecutive and non-overlapping sub-intervals. Each sub-interval corresponds to a gear position. The gear positions are assigned to each sub-interval in ascending order of their numerical values, from the 1st to the Nth gear position. N is greater than or equal to 2 and is a positive integer.

5. The method for dynamic trajectory optimization control of unmanned aerial vehicles based on reinforcement learning according to claim 3, characterized in that: Instantaneous change rate of instant reward points As a feedback signal driving the policy network to adjust the target action: The preset rate of change threshold is 0; Real-time calculation of the instantaneous rate of change of instant reward points △R(t) ; when When the value is less than 0, it is determined that the current score acquisition rate has decreased, triggering the policy network to switch between stability actions and diversity actions. when When the value is ≥0, the current target action remains unchanged.

6. The method for dynamic trajectory optimization control of unmanned aerial vehicles based on reinforcement learning according to claim 1, characterized in that: The target state guidance method includes: The updated current state vector is input into the policy network, and the policy network outputs the updated dynamic state constraint boundary. The base point region is selected based on the nearest principle: based on the current relative position features between the UAV and the target object, and using the subsequent predicted state features of the target object, the dynamic distance between the current position of the UAV and each region within the updated dynamic state constraint boundary under the target's predicted motion trajectory is calculated, and the region with the smallest dynamic distance is selected as the base point region. The policy network directly outputs the target action for controlling the UAV based on the updated current state vector, and controls the UAV to fly from the current position to the base point area. As the drone flies toward the base point area, the position of the base point area is dynamically adjusted based on the real-time updated predicted state characteristics of the target object until the drone enters the updated dynamic state constraint boundary.

7. The method for dynamic trajectory optimization control of unmanned aerial vehicles based on reinforcement learning according to claim 1, characterized in that: The ideal constraint subspace inside the dynamic state constraint boundary: The entire boundary surface of the dynamic state constraint boundary is uniformly offset inward along its normal direction by a preset distance to form a nested sub-region, thereby obtaining an ideal constraint subspace. The distance offset is dynamically adjusted according to the movement speed of the target object: when the movement speed of the target object increases, the distance offset increases accordingly, thereby expanding the ideal constraint subspace; When the target object's speed decreases, the distance offset decreases accordingly, thus shrinking the ideal constraint subspace.

8. A dynamic trajectory optimization control system for unmanned aerial vehicles based on reinforcement learning, characterized in that: The control system includes: Data acquisition module: Real-time acquisition of the target object's current state characteristics and subsequent predicted state characteristics; State construction module: Combines the relative position features, state features, and predicted state features between the UAV and the target object to form the current state vector; Action Decision and Policy Network Training Module: The current state vector is input into a pre-trained policy network, which outputs the target actions of the UAV. These target actions include mutually exclusive stable actions and diverse actions. The policy network is trained to maximize the cumulative reward score through a reward function. Calculate the instant reward points and accumulate the cumulative reward points. The stable action and the diverse action correspond to the stable action integral and the diverse action integral in the reward function, respectively. The instantaneous rate of change of the instant reward points. As a driving strategy network to adjust the feedback signal of the target action; Dynamic Constraints and Weight Adjustment Module: The dynamic integral adjustment algorithm adjusts the relative weight coefficients of the stability action integral, diversity action integral, and stability action integral in the reward function based on the target object's motion state in the current state vector. Based on the motion state of the target object, a dynamic state constraint boundary is generated to constrain the relative position features. An ideal constraint subspace is established within the dynamic state constraint boundary, and the reward function is controlled based on the relative position features and the dynamic state constraint boundary. Points for rewards within the system; The trajectory correction mechanism module updates the dynamic state constraint boundary based on the change of the current state vector, determines whether the expected target action of the UAV exceeds the dynamic state constraint boundary, and guides the UAV into the dynamic state constraint boundary through the target state guidance method when it is determined that it exceeds the dynamic state constraint boundary.

Images