A UAV optical communication link tracking method based on reinforcement learning

Abstract

This invention provides a reinforcement learning-based tracking method for UAV optical communication links, relating to the field of UAV optical communication technology. The method specifically includes: acquiring the characteristic parameters of the UAV optical communication link and constructing a UAV optical communication link channel model; defining an agent for adjusting the UAV pod attitude and establishing the agent's interaction environment based on the UAV optical communication link channel model; defining the agent's state space, action space, and reward function based on the agent's interaction environment; and training the agent using either the Deep Deterministic Policy Gradient (DDPG) algorithm or the Proximal Policy Optimization (PPO) algorithm according to the given flight mission requirements, thereby generating the optimal adjustment strategy for the UAV pod attitude. This invention, based on the DDPG algorithm and the PPO algorithm, effectively reduces the interruption rate and bit error rate of UAV optical communication in highly dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of UAV optical communication technology, and in particular to a UAV optical communication link tracking and aiming method based on reinforcement learning. Background Technology

[0002] In recent years, the rapid development of communication and information technology has driven the continuous evolution of wireless communication networks. Especially against the backdrop of increasing demands for high-speed data transmission and information security, Free Space Optical (FSO) communication has become an important research direction for future wireless communication technologies due to its high bandwidth, low interference, and high security. However, FSO technology still faces many challenges in complex environments, particularly in UAV-assisted optical communication systems, where the high-speed movement and attitude changes of the UAV, as well as external interference, lead to high communication interruption and bit error rates. Existing research mainly uses traditional control theory algorithms to optimize beam acquisition, pointing, and tracking (APT) systems. For example, PID control, Kalman filtering, and particle filtering methods have improved the control accuracy of APT systems to some extent. However, these methods still have significant drawbacks in complex environments. For instance, PID-based control methods rely on error signal feedback for adjustment, making it difficult to cope with sudden errors caused by rapid UAV maneuvers. Traditional filtering methods experience a significant decrease in filtering accuracy when faced with sudden interference, atmospheric turbulence, or obstruction, leading to communication interruptions. While methods such as particle filtering improve tracking accuracy, they are computationally expensive, and for resource-constrained UAV platforms, the real-time performance of particle filtering is difficult to meet requirements.

[0003] Traditional model-based methods primarily rely on physical modeling, mathematical optimization, and classical control theory, such as Kalman filtering and particle filtering, to achieve tracking and prediction in optical communication links. However, these methods are based on the core assumption that system dynamics and environmental interference characteristics can be accurately modeled and optimized accordingly. However, in high-dynamic environments characterized by high-speed UAV movement, severe optical signal interference, and highly unstable channel attenuation, the expressive power of the models is often limited, making it difficult to accurately characterize complex nonlinear dynamic characteristics, thus affecting the stability and reliability of optical communication links. Furthermore, constructing high-precision mathematical models typically relies on extensive prior knowledge and experimental data, and requires continuous adjustments in different application scenarios, making it difficult to meet the requirements of real-time performance and adaptability. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, this invention proposes a reinforcement learning-based tracking method for UAV optical communication links, based on the Deep Deterministic Policy Gradient (DDPG) algorithm and the Proximal Policy Optimization (PPO) algorithm, in order to reduce the interruption rate and bit error rate of UAV optical communication in highly dynamic environments.

[0005] This invention proposes a UAV optical communication link tracking and aiming method based on reinforcement learning, which includes the following process:

[0006] Obtain the characteristic parameters of the UAV optical communication link and construct the UAV optical communication link channel model;

[0007] Define an intelligent agent for adjusting the attitude of the UAV pod, and establish the interactive environment of the intelligent agent based on the UAV optical communication link channel model;

[0008] Based on the interaction environment of the agent, the state space, action space and reward function of the agent are defined respectively;

[0009] Based on the given flight mission requirements, the agent is trained using either the Depth Deterministic Policy Gradient (DDPG) algorithm or the Proximal Policy Optimization (PPO) algorithm to generate the optimal adjustment strategy for the UAV pod attitude.

[0010] Furthermore, the specific content of obtaining the characteristic parameters of the UAV optical communication link and constructing the UAV optical communication link channel model is as follows:

[0011] The characteristic parameters of the UAV optical communication link are obtained, including: UAV coordinates, user coordinates, visibility, laser wavelength, fading caused by atmospheric turbulence, and pointing alignment error;

[0012] Construct a UAV optical communication link channel model containing several links, and express the channel gain of any link in this UAV optical communication link channel model as:

[0013]

[0014] Where h k (t) represents the channel gain of the k-th link at time slot t; η is the responsivity of the user terminal; This represents the atmospheric attenuation of the k-th link at time slot t; This represents the fading of the k-th link at time slot t caused by atmospheric turbulence; This represents the directional alignment error of the k-th link at time slot t;

[0015] The atmospheric attenuation of the kth link at time slot t is calculated as follows: For time slot t, the distance between the UAV and the user is calculated using the obtained UAV coordinates and user coordinates, the attenuation coefficient is calculated using the obtained visibility and laser wavelength, and then the atmospheric attenuation of the kth link at time slot t is calculated using the Beer-Lambert law based on the calculated distance between the UAV and the user and the attenuation coefficient.

[0016] Furthermore, the specific content of the interactive environment of the intelligent agent established based on the UAV optical communication link channel model is as follows:

[0017] Simulate the beam tracking process between a drone and a user in a complex environment, establish the interaction environment of the agent, generate the state information of the agent, and update the state information of the agent according to the action selected by the agent.

[0018] The interactive environment of the intelligent agent includes three parts: a UAV optical communication link channel model, an optical imaging simulation system, and a dynamic interference simulation model. The UAV optical communication link channel model is used to feed back reward values ​​based on the state information of the intelligent agent. The optical imaging simulation system is used to map the acquired user coordinates to the camera coordinate system of the camera mounted in the UAV pod and perform distortion correction, and uses the corrected user coordinates as the user coordinates in the feature parameters of the UAV optical communication link. The dynamic interference simulation model is used to simulate different interference scenarios in complex environments.

[0019] Furthermore, the specific contents of the agent-based interaction environment, namely the agent's state space, action space, and reward function, are defined as follows:

[0020] The state space of the intelligent agent is defined as including: UAV attitude information, beam footprint pixel coordinates, and beacon point coordinates; wherein the UAV attitude information includes: pitch angle, roll angle, and yaw angle.

[0021] The action space of an intelligent agent is defined by: pod pitch angle and pod yaw angle;

[0022] Define the reward function as follows:

[0023]

[0024] Where R(t) represents the reward value of the agent at time slot t; This represents the square of the horizontal distance between the pixel coordinates of the beam footprint and the coordinates of the beacon point; This represents the square of the vertical distance between the pixel coordinates of the beam footprint and the coordinates of the beacon point.

[0025] Furthermore, the specific method for training the agent using either the Depth Deterministic Policy Gradient (DDPG) algorithm or the Proximal Policy Optimization (PPO) algorithm based on a given flight mission to generate the optimal adjustment strategy for the UAV pod attitude is as follows:

[0026] Based on the interactive environment of the intelligent agent, a 2-DOF rotating platform is used to simulate the alignment process of the UAV optical communication link;

[0027] A network architecture for training agents is constructed using the Actor-Critic framework, including an APT capturer and an APT evaluator, wherein the APT capturer includes an evaluation network and a target network; and the APT evaluator includes an evaluation network and a target network.

[0028] Based on the given flight mission, if the flight mission requires higher tracking accuracy than the response speed of the UAV pod attitude adjustment strategy, then the Deep Deterministic Policy Gradient (DDPG) algorithm is selected to train the agent; otherwise, the Proximal Policy Optimization (PPO) algorithm is selected to train the agent.

[0029] The trained agent is used to generate the optimal adjustment strategy for the attitude of the drone pod.

[0030] Furthermore, the specific method for selecting the Deep Deterministic Policy Gradient (DDPG) algorithm to train the agent is as follows:

[0031] Randomly initialize the network parameters and weights of the evaluation network in the APT capturer and the evaluation network in the APT evaluator, respectively. Copy the weights of the evaluation network in the APT capturer to the target network in the APT capturer, and copy the weights of the evaluation network in the APT evaluator to the target network in the APT capturer.

[0032] Initialize the experience replay pool, the agent's state space, and the agent's action space, and begin iterative training;

[0033] In time slot t, the agent's current state S(t) is obtained, the optimal action A(t) is selected using the evaluation network of the APT trapper, and the agent's immediate reward R(t) is calculated using the reward function. The agent's next state S(t+Δt) is updated based on the agent's immediate reward R(t), and (S(t), A(t), R(t), S(t+Δt)) is stored as a set of interaction data in the experience replay pool. The optimal action for the agent's next state S(t+Δt) is selected again using the evaluation network of the APT trapper until several sets of interaction data are stored in the experience replay pool.

[0034] Set the total number of interaction data M for each round of training iteration, and initialize the number of interaction data m for the current round of iteration. Randomly select a set of interaction data from the experience replay pool, and increment the number of interaction data for the current round of iteration by 1. Calculate the state-action value function as the agent's expected cumulative reward by maximizing the reward.

[0035] Based on the agent's expected cumulative reward, the loss function of the evaluation network of the APT evaluator is constructed, and the network parameters of the evaluation network of the APT evaluator are updated by minimizing the loss function value of the evaluation network of the APT evaluator.

[0036] According to the chain rule, the gradient of the objective function J with respect to the network parameters of the evaluation network of the APT trap is calculated, and the network parameters of the evaluation network of the APT trap are updated by gradient ascent.

[0037] Based on the updated network parameters of the APT evaluator's evaluated network and the updated network parameters of the APT capture network, a soft update mechanism is used to update the network parameters of the target network of the APT evaluator and the target network of the APT capture network.

[0038] Determine if m > M. If not, return to the step of randomly selecting a set of interaction data from the experience replay pool and continue training, and increment the amount of interaction data in the current iteration round by 1. If yes, determine if the current iteration round is greater than the preset number of training rounds. If yes, stop iterative training. If no, reinitialize the experience replay pool, the agent's state space, and the agent's action space, and start the next round of iterative training.

[0039] Furthermore, the method for selecting the optimal action A(t) using the evaluation network of the APT capture device is as follows: first, the current action is selected using the evaluation network of the APT capture device; then, noise perturbation is added to the current action using the Ornstein-Uhlenbeck process (OU) to generate the optimal action A(t) to be executed, expressed as:

[0040] A(t)=μ′(S(t))=μ(S(t)|θ μ )+N

[0041] Where μ represents the evaluation network of the APT trap; θ μ The network parameters of the evaluation network for the APT trap are represented by μ(S(t)∣θ). μ ) represents the strategy for evaluating network action selection using the APT capture device; N represents noise; μ′(S(t)) represents the strategy for evaluating network action selection using the APT capture device after adding noise perturbation.

[0042] Furthermore, the state-action value function is expressed as:

[0043] Q(S(t),A(t))=E R(t),S(t+Δt)～ E[R(S(t),A(t))+γQ(S(t+Δt),μ(S(t+Δt)))]

[0044] Where Q(S(t),A(t)) represents the Q-value of choosing the optimal action in the current state S(t), i.e., the expected cumulative reward for performing the optimal action A(t) in the current state S(t); R(S(t),A(t)) represents the immediate reward obtained after performing the optimal action A(t) in the current state S(t); γ is a discount factor, and γ∈[0,1]; Δt represents the interval time; S(t+Δt) represents the next state of the agent; Q(S(t+Δt),μ(S(t+Δt))) represents the Q-value of choosing the action according to the evaluation network μ of the APT capturer in the next state of the agent; E R(t),S(t+Δt)～ E represents the expectation of immediate reward and the next state;

[0045] The loss function of the evaluation network of the APT evaluator is expressed as:

[0046] L(θ Q )=E[(Q(S(t),A(t)∣θ Q )-y t ) 2 ]

[0047] Where L(θ) Q ) represents the loss function value of the evaluation network of the APT evaluator; E[·] represents the calculated expected value; θ Q The network parameters of the evaluation network of the APT evaluator are represented by Q(S(t),A(t)|θ). Q ) represents the Q-value predicted by the evaluation network of the APT evaluator under the current state S(t) and the optimal action A(t); y t Let represent the fitted value function of the target network of the APT evaluator, and we have:

[0048] y t =R(S(t),A(t))+γQ′(S(t+Δt),μ′(S(t+Δt)∣θ μ′ )∣θ Q′ )

[0049] Where θ Q′ θ represents the network parameters of the target network in the APT evaluator. μ′ The network parameters of the target network of the APT capturer are represented by μ′; Q′ represents the target network of the APT capturer; and μ′(S(t+Δt)∣θ) represents the target network of the APT evaluator. μ′ This indicates that the target network of the APT evaluator selects actions based on the next state;

[0050] The gradient of the objective function J with respect to the network parameters of the evaluation network of the APT trap is calculated as follows:

[0051]

[0052] in The gradient of the objective function J with respect to the network parameters of the evaluation network for the APT trapper is represented by μ(S(t)|θ). μ ) indicates that in the current state S(t), the network parameters are θ. μ The optimal action A(t) selected by the evaluation network of the APT catcher.

[0053] Furthermore, the specific method for selecting the proximal strategy to optimize the PPO algorithm for training the agent is as follows:

[0054] Randomly initialize the network parameters of the APT capturer and the APT evaluator;

[0055] Initialize the agent's state space and action space, and begin iterative training;

[0056] In time slot t, the current state S(t) of the agent is obtained, the evaluation network of the APT capturer is used to select the optimal action A(t), and the reward function is used to calculate the agent's immediate reward R(t). The next state S(t+Δt) of the agent is updated according to the agent's immediate reward R(t). (S(t), A(t)) is used as a state-action pair. The process of action selection, immediate reward calculation and state update is repeated, and several state-action pairs are saved.

[0057] For time slot t, based on the state-action pair (S(t), A(t)) at time slot t, the generalized advantage estimation (GAE) is used to construct the advantage function, and the advantage function is used to construct the loss function of the APT capturer.

[0058] The target state value is calculated based on the state-action pair (S(t), A(t)) at time slot t, and the loss function of the APT evaluator is constructed.

[0059] The network parameters of the APT capture device are updated by maximizing the loss function of the APT capture device, and the network parameters of the APT evaluator are updated by minimizing the loss function of the APT evaluator.

[0060] Extract a fixed batch of state-action pairs, update the network parameters of the APT capturer using the shearing policy gradient method according to the loss function of the APT capturer, and update the network parameters of the APT evaluator by minimizing the loss function of the APT evaluator to complete one iteration of training;

[0061] Determine if the current iteration round is greater than the preset number of training rounds. If yes, stop the iterative training. If no, reinitialize the agent's state space and action space, and start the next round of iterative training.

[0062] Furthermore, the advantage function is expressed as:

[0063]

[0064] in For the advantage predictor; δ(t) is the TD-error of time slot t; γ1 represents the discount factor; λ p δ(t+Δt) is the GAE factor; δ(t+Δt) is the TD-error from time slot t to time slot t+Δt; T represents the termination time step; δ(T-Δt) represents the TD-error from time slot T to time slot T-Δt.

[0065] The loss function of the APT capture device is expressed as:

[0066]

[0067] in This represents the total loss of the APT catcher; E represents the set of network parameters for the APT capturer; t Represents the expectation of the empirical sample; η1 is a fixed coefficient; Represents policy entropy; Let represent the loss of the pruning strategy, and we have:

[0068]

[0069] Where clip(·) is the clipping function; ∈ p It is a hyperparameter; Let be the probability ratio, and we have:

[0070]

[0071] in The strategy employed when selecting actions using the updated APT capture tool; This describes the strategy used when selecting actions using the previous APT capture method; note h. t (θ old ) = 1;

[0072] The loss function of the APT evaluator is:

[0073]

[0074] in E represents the regression loss of the APT evaluator; tRepresents the expectation of an empirical sample; Indicates the parameter φ p State-value function; This represents the value of the target state.

[0075] The beneficial effects of adopting the above technical solution are as follows:

[0076] Compared to traditional control methods such as PID and Kalman filtering, the reinforcement learning-based APT algorithm proposed in this invention has the following advantages in beam alignment tasks of the UAV FSO communication link:

[0077] (1) Improve optical signal alignment accuracy: The method of this invention uses deep reinforcement learning to dynamically adjust the beam direction, minimizing beam deviation, improving alignment accuracy, and reducing communication interruption rate. Experimental results show that the APT algorithm can reduce the interruption rate of FSO optical communication links from 40.86% to 1%. Improve signal stability.

[0078] (2) Improve data transmission quality: The method of this invention optimizes the APT scheme through reinforcement learning, controlling the bit error rate to 10%. -6 Within.

[0079] (3) Enhanced adaptability to complex environments: The method of the present invention can adaptively adjust the beam alignment strategy and improve the stability of UAV FSO communication in complex scenarios.

[0080] (4) Wide range of application scenarios: The method of the present invention can be widely applied to scenarios such as integrated air-space-ground communication, emergency rescue, and UAV relay networks, significantly improving the reliability and practicality of FSO optical communication. Attached Figure Description

[0081] Figure 1 This is a flowchart of the reinforcement learning-based APT optimization method for UAV optical communication links in this embodiment;

[0082] Figure 2 This is a flowchart illustrating the selection of the Deep Deterministic Policy Gradient (DDPG) algorithm for training the agent in this embodiment.

[0083] Figure 3 This is a flowchart illustrating the process of selecting a proximal strategy to optimize the PPO algorithm for training the agent in this embodiment. Detailed Implementation

[0084] To facilitate understanding of this application, specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. The following embodiments are illustrative of the invention but are not intended to limit its scope. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0085] Compared to traditional model-based methods, this implementation employs a model-free reinforcement learning algorithm to provide a novel solution for optimizing UAV optical communication links. Since reinforcement learning does not rely on precise mathematical modeling but instead autonomously explores optimal strategies through dynamic interaction with the environment, it can achieve efficient alignment and adaptive optimization of optical communication links in complex environments. This characteristic makes reinforcement learning methods more advantageous in nonlinear, uncertain, and rapidly changing scenarios, providing a highly robust control scheme for UAV optical communication systems.

[0086] In highly dynamic environments such as UAV optical communication links, challenges exist including difficult data acquisition, rapid environmental changes, and high real-time requirements. Therefore, there is an urgent need for an efficient reinforcement learning algorithm that can achieve precise beam alignment in complex, unstructured, and highly dynamic environments to meet the real-time application needs of UAV optical communication systems.

[0087] This embodiment provides a UAV optical communication link tracking and aiming method based on reinforcement learning, such as... Figure 1 As shown, the method includes the following steps:

[0088] Obtain the characteristic parameters of the UAV optical communication link and construct the UAV optical communication link channel model.

[0089] The specific content of obtaining the characteristic parameters of the UAV optical communication link and constructing the UAV optical communication link channel model is as follows:

[0090] The characteristic parameters of the UAV optical communication link are obtained, including: UAV coordinates, user coordinates, visibility, laser wavelength, fading caused by atmospheric turbulence, and pointing alignment error.

[0091] In this embodiment, the camera or sensor mounted on the drone pod is used to collect the pixel coordinates of the beam footprint and the beacon point coordinates in real time. For example, the pixel coordinates of the beam footprint and the beacon point coordinates at that moment can be obtained from the real-time photos taken by the camera.

[0092] Construct a UAV optical communication link channel model containing several links, and express the channel gain of any link in this UAV optical communication link channel model as:

[0093]

[0094] Where h k (t) represents the channel gain of the k-th link at time slot t; η is the responsivity of the user terminal; This represents the atmospheric attenuation of the k-th link at time slot t; This represents the fading of the k-th link at time slot t caused by atmospheric turbulence; This represents the directional alignment error of the k-th link at time slot t;

[0095] The atmospheric attenuation of the k-th link at time slot t is calculated as follows: For time slot t, the distance d between the UAV and the user is calculated using the obtained UAV coordinates and user coordinates. k ( t The attenuation coefficient β is calculated using the obtained visibility and laser wavelength, and then the distance d between the drone and the user is calculated. k ( t The atmospheric attenuation of the k-th link at time slot t is calculated using the Beer-Lambert law, along with the attenuation coefficient β, and expressed as:

[0096]

[0097] Where e is the base of the natural logarithm; d k ( t ) represents the distance between the drone and the user, calculated using the drone's coordinates and the user's coordinates; β represents the attenuation coefficient, in meters. -1 ], and there is:

[0098] β=(β dB log(10) / 10 4 [m -1 (3)

[0099] Where β dB The absorption loss is expressed in dB / km, and we have:

[0100] β dB = (3.91 / V)(λ / 550[nm]) -p [dB / km] (4)

[0101] Where 550 [nm] represents a reference wavelength of 550 nanometers; p represents the size distribution coefficient; V represents visibility; and λ represents the laser wavelength.

[0102] In this embodiment, the fading caused by atmospheric turbulence and the pointing alignment error can both be obtained using existing methods.

[0103] Define an intelligent agent for adjusting the attitude of the UAV pod, and establish the interactive environment of the intelligent agent based on the UAV optical communication link channel model.

[0104] In this embodiment, an agent is defined to generate control commands for adjusting the attitude of the UAV pod. Therefore, the agent's interaction environment is constructed around the UAV optical communication link channel model. This model reflects the physical characteristics of the optical signal propagating in the air, including channel bandwidth, receiver noise power, and path loss coefficient. The agent changes the link conditions by adjusting the pod's attitude within the interaction environment, which then provides feedback rewards based on the UAV optical communication link channel model, thereby guiding the agent to optimize its action strategy.

[0105] The specific content of the interactive environment of the intelligent agent established based on the UAV optical communication link channel model is as follows:

[0106] Simulate the beam tracking process between a drone and a user in a complex environment, establish the interaction environment of the agent, generate the agent's state information, and update the agent's state information according to the action selected by the agent.

[0107] The interactive environment of the intelligent agent includes three parts: a UAV optical communication link channel model, an optical imaging simulation system, and a dynamic interference simulation model. The UAV optical communication link channel model is used to provide feedback rewards based on the state information of the intelligent agent. The optical imaging simulation system is used to map the acquired user coordinates to the camera coordinate system of the camera mounted in the UAV pod and perform distortion correction. The corrected user coordinates are used as the user coordinates in the characteristic parameters of the UAV optical communication link.

[0108] In this embodiment, the beam tracking process of a UAV in a complex environment is simulated. An optical imaging simulation system is established, consisting of camera intrinsic parameters, camera extrinsic parameters, UAV motion trajectory data, and user motion trajectory data. This system maps user coordinates to the camera coordinate system and considers factors such as lens distortion and imaging noise to enhance the realism of visual perception.

[0109] The dynamic interference simulation model is used to simulate different interference scenarios in complex environments.

[0110] In this embodiment, the simulation environment needs to include different interference scenarios, such as atmospheric turbulence, optical signal loss caused by building obstruction, drone jitter, and high-dynamic user movement, in order to enhance the adaptability of the algorithm and improve the robustness of the optical communication link under different scenarios.

[0111] The construction method of the optical imaging simulation system is as follows: For a drone equipped with a camera in the pod, an optical imaging simulation system is established, consisting of camera intrinsic parameters, camera extrinsic parameters, drone motion trajectory data, and user motion trajectory data.

[0112] In this embodiment, the UAV trajectory data is a time-series information describing the UAV's position, attitude, and motion state in three-dimensional space, including: the UAV's coordinate position in the world coordinate system, pitch angle, roll angle, yaw angle, the UAV's velocity in the three directions, and the timestamp corresponding to each data point. The user trajectory data is a time-series information describing the target user's position and motion state in three-dimensional space, including: the user's coordinate position in the world coordinate system, velocity, and the timestamp corresponding to each data point.

[0113] Based on the agent's interaction environment, the agent's state space, action space, and reward function are defined respectively.

[0114] In this embodiment, the state space and action space of reinforcement learning are defined, and a reward function is designed to optimize the stability of the UAV optical communication link.

[0115] The agent-based interaction environment defines the specific contents of the agent's state space, action space, and reward function as follows:

[0116] The state space of the intelligent agent is defined as including: UAV attitude information, beam footprint pixel coordinates, and beacon point coordinates; wherein the UAV attitude information includes: pitch angle, roll angle, and yaw angle.

[0117] The method for obtaining the beam footprint pixel coordinates and beacon point coordinates is as follows: real-time acquisition of photos taken by a camera mounted inside the UAV pod, and extraction of the beam footprint pixel coordinates and beacon point coordinates from each photo.

[0118] In this embodiment, the state space of the agent is defined to ensure that the agent can fully perceive the state of the communication link.

[0119] The action space of an intelligent agent is defined by the pod pitch angle and the pod yaw angle.

[0120] Define the reward function as follows:

[0121]

[0122] Where R(t) represents the agent's instantaneous reward at time slot t; This represents the square of the horizontal distance between the pixel coordinates of the beam footprint and the coordinates of the beacon point; This represents the square of the vertical distance between the pixel coordinates of the beam footprint and the coordinates of the beacon point.

[0123] In this embodiment, the defined reward function reduces the bit error rate and communication interruption rate through training.

[0124] Based on the given flight mission requirements, the agent is trained using either the Deep Deterministic Policy Gradient (DDPG) algorithm or the Proximal Policy Optimization (PPO) algorithm to generate the optimal adjustment strategy for the UAV pod attitude.

[0125] The specific method for training the agent to generate the optimal adjustment strategy for the UAV pod attitude by selecting either the Depth Deterministic Policy Gradient (DDPG) algorithm or the Proximal Policy Optimization (PPO) algorithm based on the given flight mission is as follows:

[0126] Based on the interactive environment of the intelligent agent, a 2-DOF rotating platform is used to simulate the alignment process of the UAV optical communication link.

[0127] A network architecture for training agents is constructed using the Actor-Critic framework, including an APT capturer and an APT evaluator, wherein the APT capturer includes an evaluation network and a target network; and the APT evaluator includes an evaluation network and a target network.

[0128] Based on the given flight mission, if the requirement for tracking accuracy is higher than the requirement for the response speed of the UAV pod attitude adjustment strategy, then the Deep Deterministic Policy Gradient (DDPG) algorithm is selected to train the agent; otherwise, the Proximal Policy Optimization (PPO) algorithm is selected to train the agent.

[0129] In this embodiment, the Deep Deterministic Policy Gradient Algorithm (DDPG) is suitable for high-precision, low-error tasks; the Proximal Policy Optimization Algorithm (PPO) is suitable for tasks where the environment changes rapidly and policies need to be adjusted quickly.

[0130] In this embodiment, such as Figure 2As shown, APT reinforcement learning training is performed based on the DDPG algorithm. Specifically, the Deep Deterministic Policy Gradient (DDPG) algorithm is used to optimize the APT process of the UAV optical communication link. Leveraging the decision-making capabilities of reinforcement learning in a continuous action space, the optical communication link can adaptively adjust in a highly dynamic environment. The DDPG training process consists of an agent, an environment model, state inputs, action outputs, and reward feedback. The state space includes beam footprint pixel coordinates, UAV attitude, and beacon point coordinates, while the action space covers pod pitch and yaw angles. The agent, through the policy network (APT catcher), makes control commands based on the current state and adjusts the optical axis direction of the optical communication link. The environment updates its state upon receiving the action and calculates an immediate reward value based on the image pixel coordinates and the communication bit error rate. Higher rewards indicate a more stable policy, while penalties are applied when the bit error rate increases or the link is interrupted to optimize the decision direction. The value network (APT evaluator) evaluates the long-term reward of the current action, guiding policy convergence and enabling the agent to gradually learn the optimal beam tracking strategy. During training, DDPG utilizes experience replay to reduce data correlation and employs a target network for stable updates to prevent policy convergence instability. Furthermore, the network parameters of the APT capturer are gradually adjusted through soft updates to improve learning stability and enhance the algorithm's adaptability in highly dynamic environments. Through this closed-loop reinforcement learning framework, the agent can progressively optimize the beam alignment strategy of the optical communication link, minimizing the bit error rate, reducing the link interruption rate, and improving the robustness of the communication system.

[0131] The specific method for training the agent using the Depth Deterministic Policy Gradient (DDPG) algorithm is as follows:

[0132] Randomly initialize the network parameters and weights of the evaluation network in the APT capturer and the evaluation network in the APT evaluator, respectively. Copy the weights of the evaluation network in the APT capturer to the target network in the APT capturer, and copy the weights of the evaluation network in the APT evaluator to the target network in the APT capturer.

[0133] Initialize the experience replay pool, the agent's state space, and the agent's action space.

[0134] In time slot t, the agent's current state S(t) is obtained, the optimal action A(t) is selected using the evaluation network of the APT trapper, and the agent's immediate reward R(t) is calculated using the reward function. The agent's next state S(t+Δt) is updated based on the agent's immediate reward R(t), and (S(t), A(t), R(t), S(t+Δt)) is stored as a set of interaction data in the experience replay pool. The optimal action for the agent's next state S(t+Δt) is selected again using the evaluation network of the APT trapper until several sets of interaction data are stored in the experience replay pool.

[0135] The method for selecting the optimal action A(t) using the evaluation network of the APT capture device is as follows: First, the current action is selected using the evaluation network of the APT capture device. Then, noise perturbation is added to the current action using the Ornstein-Uhlenbeck (OU) process to generate the optimal action A(t) to be executed, expressed as:

[0136] A(t)=μ′(S(t))=μ(S(t)|θ μ )+N (6)

[0137] Where μ represents the evaluation network of the APT trap; θ μ The network parameters of the evaluation network for the APT trap are represented by μ(S(t)∣θ). μ ) represents the strategy for evaluating network action selection using the APT capture device; N represents noise; μ′(S(t)) represents the strategy for evaluating network action selection using the APT capture device after adding noise perturbation.

[0138] In this embodiment, in order to enhance the strategy exploration capability, an Ornstein-Uhlenbeck (OU) process is introduced to perturb the actions.

[0139] In this implementation, the parameters of the APT capturer and APT evaluator are randomly initialized. The target network is initialized to synchronize its parameters with the current network. The experience pool is cleared to prepare for storing sampled data. The environmental state is collected, and an action is selected according to the policy. The action is executed, a reward is obtained, and the environmental state is updated. Experience data is stored and mini-batch sampling is performed. An experience replay mechanism is used to enable the agent to learn effective policies from historical data, improving training stability. Simultaneously, the target network is used for stable policy updates to reduce fluctuations during training, and soft updates are used to reduce model oscillations and improve learning efficiency. Soft updates are used to adjust the parameters of the target network until the average reward value reaches the expected value, at which point training stops.

[0140] Set the total number of interaction data M for each round of training iteration, and initialize the number of interaction data m for the current iteration round. Randomly select a set of interaction data from the experience replay pool, and increment the number of interaction data for the current iteration round by 1. Calculate the state-action value function as the agent's expected cumulative reward by maximizing the reward.

[0141] The state-action value function is expressed as follows:

[0142] Q(S(t),A(t))=E R(t),S(t+Δt)～ E[R(S(t),A(t))+γQ(S(t+Δt),μ(S(t+Δt)))] (7)

[0143] Where Q(S(t),A(t)) represents the Q-value of choosing the optimal action in the current state S(t), i.e., the expected cumulative reward for performing the optimal action A(t) in the current state S(t); R(S(t),A(t)) represents the immediate reward obtained after performing the optimal action A(t) in the current state S(t); γ is a discount factor, and γ∈[0,1]; Δt represents the interval time; S(t+Δt) represents the next state of the agent; Q(S(t+Δt),μ(S(t+Δt))) represents the Q-value of choosing the action according to the evaluation network μ of the APT capturer in the next state of the agent; E R(t),S(t+Δt)～ E represents the expectation of the immediate reward and the next state, which is the average of the two.

[0144] Based on the agent's expected cumulative reward, the loss function of the evaluation network of the APT evaluator is constructed, and the network parameters of the evaluation network of the APT evaluator are updated by minimizing the loss function value of the evaluation network of the APT evaluator.

[0145] The loss function of the evaluation network of the APT evaluator is expressed as:

[0146] L(θ Q )=E[(Q(S(t),A(t)∣θ Q )-y t ) 2 (8)

[0147] Where L(θ) Q ) represents the loss function value of the evaluation network of the APT evaluator; E[·] represents the calculated expected value; θ Q The network parameters of the evaluation network of the APT evaluator are represented by Q(S(t),A(t)|θ). Q ) represents the Q-value predicted by the evaluation network of the APT evaluator under the current state S(t) and the optimal action A(t); y t This represents the fitted value function of the target network of the APT evaluator, although y t It also depends on θ Q However, this is usually overlooked, and there are:

[0148] y t =R(S(t),A(t))+γQ′(S(t+Δt),μ′(S(t+Δt)∣θ μ′ )∣θ Q′ (9)

[0149] Where θ Q′ θ represents the network parameters of the target network in the APT evaluator. μ′The network parameters of the target network of the APT capturer are represented by μ′; Q′ represents the target network of the APT capturer; and μ′(S(t+Δt)∣θ) represents the target network of the APT evaluator. μ′ ) indicates that the target network of the APT evaluator selects actions based on the next state.

[0150] According to the chain rule, the gradient of the objective function J with respect to the network parameters of the evaluation network of the APT trap is calculated, and the network parameters of the evaluation network of the APT trap are updated by gradient ascent.

[0151] The gradient of the objective function J with respect to the network parameters of the evaluation network of the APT trap is calculated as follows:

[0152]

[0153] in The gradient of the objective function J with respect to the network parameters of the evaluation network for the APT trapper is represented by μ(S(t)|θ). μ ) indicates that in the current state S(t), the network parameters are θ. μ The optimal action A(t) selected by the evaluation network of the APT catcher.

[0154] In this embodiment, for time slot t, the APT catcher can be updated to the expected return of the APT catcher parameters from the initial distribution according to the chain rule, that is, the network parameters of the APT catcher are optimized by using the policy gradient theorem to maximize the long-term expected return J.

[0155] Based on the updated network parameters of the APT evaluator's evaluated network and the updated network parameters of the APT capture network, a soft update mechanism is used to update the network parameters of the target network of the APT evaluator and the target network of the APT capture network.

[0156] In this embodiment, in order to reduce the frequent updates of online network parameters, DDPG adopts a soft update strategy and stores empirical data.

[0157] The method for updating the network parameters of the target network of the APT evaluator and the target network of the APT catcher using a soft update mechanism is as follows:

[0158]

[0159] Where θ μ0 θ represents the updated network parameters of the target network of the APT capture device. Q0 This represents the updated network parameters of the target network in the APT evaluator.

[0160] Determine if m > M. If not, return to the step of randomly selecting a set of interaction data from the experience replay pool and continue training, and increment the amount of interaction data in the current iteration round by 1. If yes, determine if the current iteration round is greater than the preset number of training rounds. If yes, stop iterative training. If no, reinitialize the experience replay pool, the agent's state space, and the agent's action space, and start the next round of iterative training.

[0161] Through the training process described above, the DDPG agent gradually learns to output alignment control actions that maximize rewards under different states, reducing the bit error rate and interruption rate. Simulation results show that the DDPG algorithm can effectively solve the alignment and tracking problem between the UAV and the ground user. During training, the total reward value gradually increases and tends to stabilize, the cumulative reward curve is relatively smooth, and the convergence performance is good. Compared with other algorithms, the strategy obtained by DDPG exhibits higher link stability in high-speed motion environments.

[0162] This implementation uses a proximal policy optimization algorithm to achieve APT control. The PPO algorithm also employs the Actor-Critic framework, but its policy network directly outputs the action probability distribution, making it suitable for both discrete and continuous actions. It models continuous actions using a Gaussian distribution, and the value network estimates the state value. The PPO agent structure is similar to that of DDPG, but the method for updating the policy differs. For example... Figure 3 As shown, the training process of the APT algorithm based on PPO includes the following loop: First, initialize the parameters of the policy network and value network, as well as the environmental state; then repeatedly perform the following operations until convergence: (1) Interact with the environment to collect the state, action, and reward sequences of a certain time step and save the trajectory data; (2) In each round of updates, calculate the advantage estimate and target state value of each time step based on the collected data; (3) Use the above data to perform gradient updates on the value network and policy network respectively, iterating multiple times: the value network minimizes the state value loss, and the policy network updates the parameters using the shearing policy gradient method shown in the formula; (4) After updating a certain batch of data, enter the next round of interaction with the environment. Through this near-end policy constraint update method, PPO can gradually improve policy performance and maintain stable convergence of training. Simulation results show that the PPO algorithm can also effectively realize the optical link alignment control between the UAV and the terminal, and its cumulative reward increases continuously with training iteration. Compared with DDPG, PPO may have slightly larger fluctuations in the early stage of training, but after appropriately adjusting the batch size, learning rate, and other hyperparameters, it can also achieve good convergence and link stability improvement.

[0163] The specific method for selecting a proximal strategy to optimize the PPO algorithm for training agents is as follows:

[0164] Randomly initialize the network parameters of the APT capturer and the APT evaluator.

[0165] Initialize the agent's state space and action space, and begin iterative training.

[0166] In time slot t, the agent's current state S(t) is obtained, the optimal action A(t) is selected using the evaluation network of the APT capture device, and the agent's immediate reward R(t) is calculated using the reward function. The agent's next state S(t+Δt) is updated based on the agent's immediate reward R(t). The process of action selection, immediate reward calculation and state update is repeated, and several sets of state-action pairs are saved.

[0167] In this implementation, the evaluation network selects an action to execute based on the current policy, i.e., the APT capture device. The action is executed, and a reward and a new state are obtained. State transition data, i.e., state-action pairs, are stored for training.

[0168] For time slot t, based on the state-action pair (S(t), A(t)) at time slot t, the generalized advantage estimation (GAE) is used to construct the advantage function, which is expressed as:

[0169]

[0170] in For the advantage predictor; δ(t) is the TD-error of time slot t; γ1 represents the discount factor; λ p δ(t+Δt) is the GAE factor; δ(t+Δt) is the TD-error from time slot t to time slot t+Δt; T represents the termination time step; δ(T-Δt) represents the TD-error from T to time slot T-Δt.

[0171] In this implementation, Generalized Advantage Estimation (GAE) is used to optimize the policy gradient, improving the stability of beam tracking. The state value function is calculated, and the policy network is dynamically updated to enhance the agent's adaptability to different optical signal environments, ensuring that reinforcement learning can operate stably in complex optical communication channels.

[0172] The loss function of the APT trap, which consists of the pruning policy loss and the entropy reward, is constructed using the advantage function and expressed as follows:

[0173]

[0174] in This represents the total loss of the APT catcher; E represents the set of network parameters for the APT capturer; t η1 represents the expected value of the empirical sample, which is the average of all time steps in the current batch of data; η1 is a fixed coefficient. This represents the addition of policy entropy, primarily used for agent action exploration to prevent entering local optima; This represents the loss from the pruning strategy, used for more effective strategy updates, and includes:

[0175]

[0176] Where clip(·) is the clipping function; ∈ p As a hyperparameter, it is usually set to ∈ p =0.2; Let be the probability ratio, and we have:

[0177]

[0178] in The strategy employed when selecting actions using the updated APT capture tool; This describes the strategy used when selecting actions using the previous APT capture method; note h. t (θ old ) = 1.

[0179] In this embodiment, the second term in expression (14) limits the large difference between the new policy update and the old policy update.

[0180] The target state value is calculated based on the state-action pair (S(t), A(t)) at time slot t, and the loss function of the APT evaluator is constructed as follows:

[0181]

[0182] in E represents the regression loss of the APT evaluator; t It represents the expectation of the empirical sample, which is the average of all time steps in the current batch of data; Indicates the parameter φ p State-value function; This represents the value of the target state.

[0183] Extract a fixed batch of state-action pairs, update the network parameters of the APT capturer using the shearing policy gradient method according to the loss function of the APT capturer, and simultaneously update the network parameters of the APT evaluator by minimizing the loss function of the APT evaluator, thus completing one iteration of training.

[0184] Determine whether the current iteration round is greater than the preset number of training rounds or whether the current training process meets the convergence condition. If the current iteration round is greater than the preset number of training rounds or the current training process meets the convergence condition, stop the iterative training. If not, reinitialize the agent's state space and action space, and start the next round of iterative training.

[0185] The convergence condition is that the loss function of the APT catcher converges or the loss function of the APT evaluator converges.

[0186] In this implementation, the training process needs to be terminated when the loss function converges and the policy update can no longer significantly improve performance. Through the above training process, the PPO agent gradually learns how to select the optimal beam alignment strategy under different states to maximize cumulative rewards. Simulation results show that the PPO algorithm can effectively improve the alignment accuracy and link stability of the UAV optical communication link, and has good robustness and convergence in high-speed motion environments.

[0187] The trained agent is used to generate the optimal adjustment strategy for the attitude of the drone pod.

[0188] In this embodiment, for a trained agent, the state of the agent is acquired in real time and input into the trained agent, the selected optimal action is output, and the optimal action is used as the best adjustment strategy for the attitude of the UAV pod.

[0189] In this implementation, two reinforcement learning algorithms, DDPG and PPO, are used for comparative training to evaluate the impact of different optimization methods on APT performance and to select the most suitable algorithm for different requirements in real-world application scenarios. Specifically, the trained agent is applied to an actual UAV optical communication pod platform, and parameters are adjusted and optimized based on real-time flight data. Through continuous learning in a real environment, the model can gradually adapt to external interference and complex flight conditions, improving the success rate of aligning with the user and establishing a communication link.

[0190] The PPO algorithm was validated in both simulation and real-world testing environments. The test results show that the PPO algorithm exhibits a significant advantage in rapid adaptation, with an average convergence time of 8 × 10⁻⁶ steps. 4 Steps, compared to the DDPG algorithm's average convergence steps of 1.5 × 10⁻⁶. 5 The number of steps was reduced by 47%; its sample utilization rate was 68%, nearly twice that of the DDPG algorithm's 35%. Especially in dynamic environments, such as sudden turbulence, it could complete policy adjustments in just 12 seconds. However, its control accuracy (0.25±0.12 mrad) showed an error increase of approximately 52% compared to the DDPG algorithm's control accuracy (0.12±0.03 mrad). Therefore, when high accuracy and low error are required for flight missions, the DDPG algorithm should be chosen for agent training; when the flight mission environment changes rapidly and requires quick policy adjustments, the PPO algorithm should be chosen for agent training.

[0191] Furthermore, the algorithm's performance was verified in simulation and real-world testing environments, and the stability and reliability of the intelligent agent in high-dynamic environments were tested through UAV optical communication experiments. This implementation method was compared with traditional APT schemes such as PID control and Kalman filtering to analyze the advantages of reinforcement learning in high-dynamic optical communication, ensuring the effectiveness and feasibility of the designed intelligent control algorithm in practical applications. Test results show that PID control can provide stable control in low-dynamic environments, such as static base station-to-UAV links, but it struggles to adapt to complex disturbances in high-dynamic environments. Kalman filtering can provide relatively good state estimation in low-noise environments, but its tracking accuracy is limited in UAV optical communication systems with strong nonlinearity and drastic environmental changes. In contrast, DDPG and PPO can automatically adapt to environmental changes based on reinforcement learning, exhibiting superior stability and accuracy in high-dynamic scenarios, providing a more robust intelligent APT optimization scheme for UAV optical communication links.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A tracking and aiming method for UAV optical communication links based on reinforcement learning, characterized in that, This method includes the following steps: The characteristic parameters of the UAV optical communication link are obtained and a channel model of the UAV optical communication link is constructed. The specific content is as follows: The characteristic parameters of the UAV optical communication link are obtained, including: UAV coordinates, user coordinates, visibility, laser wavelength, fading caused by atmospheric turbulence, and pointing alignment error; Construct a UAV optical communication link channel model containing several links, and express the channel gain of any link in this UAV optical communication link channel model as: ； in This represents the channel gain of the k-th link at time slot t; For user-side responsiveness; This represents the atmospheric attenuation of the k-th link at time slot t; This represents the fading of the k-th link at time slot t caused by atmospheric turbulence; This represents the directional alignment error of the k-th link at time slot t; The atmospheric attenuation of the kth link at time slot t is calculated as follows: For time slot t, the distance between the UAV and the user is calculated using the obtained UAV coordinates and user coordinates, the attenuation coefficient is calculated using the obtained visibility and laser wavelength, and then the atmospheric attenuation of the kth link at time slot t is calculated using the Beer-Lambert law based on the calculated distance between the UAV and the user and the attenuation coefficient. Define an agent for adjusting the attitude of the UAV pod, and establish the agent's interaction environment based on the UAV optical communication link channel model. The specific content is as follows: Simulate the beam tracking process between a drone and a user in a complex environment, establish the interaction environment of the agent, generate the state information of the agent, and update the state information of the agent according to the action selected by the agent. The interactive environment of the intelligent agent comprises three parts: a UAV optical communication link channel model, an optical imaging simulation system, and a dynamic interference simulation model. The UAV optical communication link channel model is used to feed back reward values ​​based on the intelligent agent's state information. The optical imaging simulation system maps the acquired user coordinates to the camera coordinate system of the camera mounted in the UAV pod and performs distortion correction, using the corrected user coordinates as the user coordinates in the characteristic parameters of the UAV optical communication link. The dynamic interference simulation model simulates different interference scenarios under complex environments. Based on the agent's interaction environment, the agent's state space, action space, and reward function are defined, as follows: The state space of the intelligent agent is defined as including: UAV attitude information, beam footprint pixel coordinates, and beacon point coordinates; wherein the UAV attitude information includes: pitch angle, roll angle, and yaw angle. The action space of an intelligent agent is defined by: pod pitch angle and pod yaw angle; Define the reward function as follows: ； in This represents the agent's reward value at time slot t; This represents the square of the horizontal distance between the pixel coordinates of the beam footprint and the coordinates of the beacon point; This represents the square of the vertical distance between the pixel coordinates of the beam footprint and the coordinates of the beacon point; Based on the given flight mission requirements, the agent is trained using either the Deep Deterministic Policy Gradient (DDPG) algorithm or the Proximal Policy Optimization (PPO) algorithm to generate the optimal adjustment strategy for the UAV pod attitude.

2. The UAV optical communication link tracking and aiming method based on reinforcement learning according to claim 1, characterized in that, The specific method for training the agent to generate the optimal adjustment strategy for the UAV pod attitude by selecting either the Depth Deterministic Policy Gradient (DDPG) algorithm or the Proximal Policy Optimization (PPO) algorithm based on the given flight mission is as follows: Based on the interactive environment of the intelligent agent, a 2-DOF rotating platform is used to simulate the alignment process of the UAV optical communication link; A network architecture for training agents is constructed using the Actor-Critic framework, including an APT capturer and an APT evaluator, wherein the APT capturer includes an evaluation network and a target network; and the APT evaluator includes an evaluation network and a target network. Based on the given flight mission, if the flight mission requires higher tracking accuracy than the response speed of the UAV pod attitude adjustment strategy, then the Deep Deterministic Policy Gradient (DDPG) algorithm is selected to train the agent; otherwise, the Proximal Policy Optimization (PPO) algorithm is selected to train the agent. The trained agent is used to generate the optimal adjustment strategy for the attitude of the drone pod.

3. The UAV optical communication link tracking and aiming method based on reinforcement learning according to claim 2, characterized in that, The specific method for training the agent using the Depth Deterministic Policy Gradient (DDPG) algorithm is as follows: Randomly initialize the network parameters and weights of the evaluation network in the APT capturer and the evaluation network in the APT evaluator, respectively. Copy the weights of the evaluation network in the APT capturer to the target network in the APT capturer, and copy the weights of the evaluation network in the APT evaluator to the target network in the APT capturer. Initialize the experience replay pool, the agent's state space, and the agent's action space, and begin iterative training; In the time slot Get the current state of the agent The optimal action is selected using the evaluation network of the APT capture device. And use the reward function to calculate the agent's immediate reward. Based on the agent's immediate reward Update the agent's next state , and then Stored as a set of interactive data in the experience replay pool; The evaluation network of the APT trap continues to be used to select the agent's next state. The optimal action is determined until several sets of interaction data are stored in the experience replay pool. Set the total number of interaction data M for each round of training iteration, and initialize the number of interaction data m for the current round of iteration. Randomly select a set of interaction data from the experience replay pool, and increment the number of interaction data for the current round of iteration by 1. Calculate the state-action value function as the agent's expected cumulative reward by maximizing the reward. Based on the agent's expected cumulative reward, the loss function of the evaluation network of the APT evaluator is constructed, and the network parameters of the evaluation network of the APT evaluator are updated by minimizing the loss function value of the evaluation network of the APT evaluator. According to the chain rule, the gradient of the objective function J with respect to the network parameters of the evaluation network of the APT trap is calculated, and the network parameters of the evaluation network of the APT trap are updated by gradient ascent. Based on the updated network parameters of the APT evaluator's evaluated network and the updated network parameters of the APT capture network, a soft update mechanism is used to update the network parameters of the target network of the APT evaluator and the target network of the APT capture network. Determine if m > M is true. If not, return to the step of randomly selecting a set of interaction data from the experience replay pool and continue training, and increment the amount of interaction data in the current iteration round by 1. If yes, determine if the current iteration round is greater than the preset number of training rounds. If yes, stop iterative training. If not, the experience replay pool, the agent's state space, and the agent's action space are reinitialized, and the next round of iterative training begins.

4. The UAV optical communication link tracking and aiming method based on reinforcement learning according to claim 3, characterized in that, The evaluation network using the APT trap selects the optimal action. The method is as follows: First, the evaluation network of the APT capture device is used to select the current action. Then, the Ornstein-Uhlenbeck process (OU) is used to add noise perturbation to the current action to generate the optimal action to be executed. , represented as: ； in This represents the evaluation network for the APT capture device; This represents the network parameters of the evaluation network for the APT capture device; This indicates the strategy for evaluating network action selection using the APT capturer; Indicates noise; This represents the strategy for evaluating the network's action selection using the APT capturer after adding noise perturbations.

5. The UAV optical communication link tracking and aiming method based on reinforcement learning according to claim 4, characterized in that, The state-action value function is expressed as follows: ； in Indicates the current state The Q-value for selecting the optimal action in the current state is used to evaluate the current action. Execute the optimal action Expected cumulative rewards; Indicates the current state Execute the optimal action The instant reward obtained afterward; It is a discount factor, and ; Indicates the interval time; Indicates the next state of the agent; This indicates the agent's next state, based on the evaluation network of the APT trapper. The Q value for selecting an action; This indicates expectations for immediate rewards and the next state; The loss function of the evaluation network of the APT evaluator is expressed as: ； in This represents the loss function value of the evaluation network of the APT evaluator; This indicates the calculation of the expected value; This represents the network parameters of the evaluation network in the APT evaluator; This represents the Q-value predicted by the evaluation network of the APT evaluator under the current state S(t) and the optimal action A(t); Let represent the fitted value function of the target network of the APT evaluator, and we have: ； in This represents the network parameters of the target network in the APT evaluator; This represents the network parameters of the target network for the APT capturer; The target network represents the APT capture device; This represents the target network of the APT evaluator; This indicates that the target network of the APT evaluator selects actions based on the next state; The gradient of the objective function J with respect to the network parameters of the evaluation network of the APT trap is calculated as follows: ； in This represents the gradient of the objective function J with respect to the network parameters of the evaluation network for the APT trap; Indicates the current state Below, the network parameters are: The optimal action selected by the evaluation network of the APT capture device .

6. The UAV optical communication link tracking and aiming method based on reinforcement learning according to claim 5, characterized in that, The specific method for selecting a proximal strategy to optimize the PPO algorithm for training agents is as follows: Randomly initialize the network parameters of the APT capturer and the APT evaluator; Initialize the agent's state space and action space, and begin iterative training; In the time slot Get the current state of the agent The optimal action is selected using the evaluation network of the APT capture device. And use the reward function to calculate the agent's immediate reward. Based on the agent's immediate reward Update the agent's next state , and As a set of state-action pairs, repeat the process of action selection, instant reward calculation and state update, and save several sets of state-action pairs; For time slot t, the action is based on the state at time slot t. The generalized advantage estimation (GAE) is used to construct the advantage function, and the loss function of the APT trap is constructed using the advantage function. Based on the state and action at time slot t Calculate the target state value and construct the loss function for the APT evaluator; The network parameters of the APT capture device are updated by maximizing the loss function of the APT capture device, and the network parameters of the APT evaluator are updated by minimizing the loss function of the APT evaluator. Extract a fixed batch of state-action pairs, update the network parameters of the APT capturer using the shearing policy gradient method according to the loss function of the APT capturer, and update the network parameters of the APT evaluator by minimizing the loss function of the APT evaluator to complete one iteration of training; Determine if the current iteration round is greater than the preset number of training rounds. If yes, stop the iterative training. If no, reinitialize the agent's state space and action space, and start the next round of iterative training.

7. The UAV optical communication link tracking and aiming method based on reinforcement learning according to claim 6, characterized in that, The advantage function is expressed as: ； in For advantage predictor; The TD-error for time slot t; Indicates the discount factor; GAE factor; From time slot t to time slot TD-error; Indicates the termination time step; express Time slot TD-error; The loss function of the APT capture device is expressed as: ； in This represents the total loss of the APT catcher; This represents the set of network parameters for the APT capturer; Represents the expectation of an empirical sample; It is a fixed coefficient; Represents policy entropy; Let represent the loss of the pruning strategy, and we have: ； in This is a pruning function; It is a hyperparameter; Let be the probability ratio, and we have: ； in The strategy employed when selecting actions using the updated APT capture tool; This describes the strategy used when selecting actions using the previous APT capture method; note... ; The loss function of the APT evaluator is: ； in This represents the regression loss of the APT evaluator; Represents the expectation of an empirical sample; Indicates parameters State-value function; This represents the value of the target state.

Images