Track optimization method for relay unmanned aerial vehicle

Abstract

The invention discloses a relay unmanned aerial vehicle track optimization method, which comprises the following steps of: firstly, constructing a lossy relay unmanned aerial vehicle communication system model under finite length coding, and covering communication time slot division, unmanned aerial vehicle flight constraint and a channel model; deducing outage probability expressions of two communication strategies under finite length coding, wherein the outage probability expressions are an independent decoding strategy and a joint decoding strategy respectively; then, a trajectory optimization problem is converted into a Markov decision process (MDP), and a state space, an action space, a reward function and a state transition mechanism are clarified; solving by adopting a joint optimization algorithm based on depth deterministic strategy gradient (DDPG) to realize joint optimization of the unmanned aerial vehicle track and the communication strategy; and finally, outputting an optimal unmanned aerial vehicle track and strategy configuration to provide reference for actual system deployment. The method can effectively reduce the outage probability of the system and improve the communication reliability under the condition of limited length coding, and is suitable for various scenes in 6G and future air-ground cooperative communication networks.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, specifically relating to a method for optimizing the trajectory of a relay drone. Background Technology

[0002] With the deepening research into sixth-generation (6G) mobile communication, air-ground cooperative networks have become an important development direction for improving system robustness and coverage. Unmanned aerial vehicles (UAVs), with their advantages of rapid deployment, high mobility, and low cost, are widely regarded as ideal carriers for constructing temporary relay links, especially suitable for emergency communication recovery in natural disasters, large-scale events, or remote areas. However, most existing research on UAV relays is based on the idealized assumption of infinitely long coding, which assumes that the system can infinitely approach Shannon capacity, thus ignoring the reality of finite-length coding due to latency, energy consumption, or hardware resource limitations in actual communication. Under this condition, the traditional capacity formula based on the Shannon limit is no longer applicable, system reliability decreases significantly, and the probability of communication interruption increases accordingly.

[0003] On the other hand, existing UAV trajectory optimization efforts mainly focus on lossless transmission or classic relay protocols, such as Amplify Forward (AF) and Decode Forward (DF). While AF is simple to implement, it amplifies noise under poor channel conditions; DF, while suppressing noise, requires relay nodes to decode perfectly, and data is discarded if decoding fails, resulting in resource waste. In recent years, Lossy Forward (LF) protocols have attracted attention because they allow relays to continue forwarding information despite some distortion, making them particularly suitable for AI-oriented communication scenarios. As long as the AI ​​makes the correct decision, the lossy recovery of information can still be effectively utilized. However, system modeling, outage probability characterization, and trajectory optimization methods for LF protocols under finite-length coding conditions are still lacking.

[0004] Therefore, there is an urgent need for a trajectory optimization method that can jointly consider the characteristics of the LF protocol and the maneuverability of UAVs under finite length coding constraints, so as to minimize the probability of communication interruption and improve system reliability in resource-constrained environments. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a method for optimizing the trajectory of a relay UAV. First, a lossy relay UAV communication system model under finite-length coding is constructed, encompassing communication time slot partitioning, UAV flight constraints, and channel models. Next, the outage probability expressions for two communication strategies under finite-length coding are derived: independent decoding and joint decoding strategies. Then, the trajectory optimization problem is transformed into a Markov Decision Process (MDP), clearly defining the state space, action space, reward function, and state transition mechanism. Subsequently, a joint optimization algorithm based on Deep Deterministic Policy Gradient (DDPG) is used to solve the problem, achieving joint optimization of the UAV trajectory and communication strategy. Finally, the optimal UAV trajectory and strategy configuration are output, providing a reference for practical system deployment. This invention can effectively reduce the system outage probability and improve communication reliability under finite-length coding conditions, and is applicable to various scenarios in 6G and future air-to-ground cooperative communication networks.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] Step 1: Construct a lossy relay UAV communication system model under finite-length coding, including communication time slot partitioning, UAV flight constraints, and channel model;

[0008] Step 2: Derive the interruption probability expressions for two communication strategies under finite-length coding, namely independent decoding and joint decoding strategies;

[0009] Step 3: Transform the trajectory optimization problem into a Markov Decision Process (MDP), and define the state space, action space, reward function, and state transition mechanism;

[0010] Step 4: Solve the problem using a joint optimization algorithm based on Deep Deterministic Policy Gradient (DDPG) to achieve joint optimization of UAV trajectory and communication strategy;

[0011] Step 5: Output the optimal UAV trajectory and strategy configuration to provide a reference for actual system deployment.

[0012] Furthermore, step 1 specifically includes:

[0013] Step 1-1: Construct a model of a lossy cooperative UAV relay communication system under a finite encoding length scenario;

[0014] The drone is responsible for transmitting information to K>1 users on the ground, where the user set is represented as follows:

[0015] Assume the communication process occurs within a finite time interval. Where T>0; the flight time T of the UAV is divided into N>1 time slots, and the set of time slots is represented as follows. t represents the time slot; during the communication process, the served users use frequency division multiple access for multi-user access;

[0016] Let R s0 R s1 R s2 These represent the coding rates of the base station to UAV link, base station to user link, and UAV to user link, respectively, and k0, k1, and k2 represent the information sequence length of each link.

[0017] In three-dimensional coordinates, the position of ground user k at time slot t is represented as follows: The coordinates of the UAV are represented as q[t] = (x q [t],y q [t],z q ), z q It is a constant. Let x and y represent the coordinates of ground user k at time slot t, respectively. q [t],y q [t],z q Let x, y, and z represent the coordinates of the UAV at time slot t, respectively, and let q be the initial position of the UAV. I =(x I ,y I ,z q In the communication model, the behavior of drones is constrained by the following conditions:

[0018] q[1]=q I

[0019] ||q[t+1]-q[t]||≤v max

[0020] In the formula, v max The maximum flight speed of the drone in the horizontal direction is represented by Δq[t] = (Δx) which limits the change in the drone's coordinates. q [t],Δy q [t],0) is used to achieve this; q[t+1] represents the coordinate representation of the UAV in time slot t+1;

[0021] Step 1-2: Construct a lossy relay UAV communication channel model under finite-length coding;

[0022] Channel modeling is performed using a probabilistic line-of-sight channel model based on elevation angle. In the communication link, the line-of-sight link probabilities at the transmitting and receiving ends are... Non-line-of-sight link probability Represented as:

[0023]

[0024] In the formula, For environmental parameters, θ t,r The elevation angle between the transmitting and receiving ends of the link is expressed as:

[0025]

[0026] In the formula, ΔH represents the height difference between the transmitting and receiving ends of the link, and d t,r The distance between the sending and receiving ends of the link can be calculated using the following formula:

[0027]

[0028] Where, x t y t z t These represent the x-axis, y-axis, and z-axis coordinates of the transmitting end, respectively. r y r z r These represent the x-axis, y-axis, and z-axis coordinates of the receiving end, respectively.

[0029] At this point, the path losses for line-of-sight links and non-line-of-sight links are expressed as follows:

[0030]

[0031] In the formula, f represents the carrier frequency, and η Los η NLoS Indicates additional path loss; η Los Representing a line-of-sight scenario, η NLoS Indicates a non-line-of-sight scene;

[0032] Using the average path loss as the path loss between the receiver and transmitter of each link, the average path loss PL between the UAV and the k-th user is... k Represented as:

[0033]

[0034] Assume the given base station transmit power is P b Given the UAV's transmit power P UAV Then the signal power received by the k-th user from the base station and from the drone and They are represented as follows:

[0035]

[0036] In the formula, |g| 2 Indicates channel fading gain;

[0037] The instantaneous signal-to-noise ratio of each link between the k-th user and the base station and the drone is expressed as:

[0038]

[0039] In the formula, N0 represents the noise power spectral density of additive Gaussian noise; γ b,k γ represents the instantaneous signal-to-noise ratio of the link between the k-th user and the base station. UAV,k This represents the instantaneous signal-to-noise ratio of the link between the k-th user and the drone.

[0040] Furthermore, step 2 specifically includes:

[0041] Step 2-1: For the lossy cooperative UAV relay communication system model established in Step 1, consider three scenarios in which communication will not be interrupted;

[0042] (1) Considering that the base station to user link is independently decoded and no frame errors occur, the interruption probability under this case is denoted as P1;

[0043] (2) Considering the case where frame errors occur in the base station to user link, the base station to UAV link and the UAV to user link are decoded independently and no frame errors occur. The interruption probability in this case is denoted as P2.

[0044] (3) Considering that the base station to UAV link is decoded independently and no frame errors occur, the base station to user and UAV to user links are decoded jointly to achieve no frame errors. The interruption probability in this case is denoted as P3.

[0045] Step 2-2: Assume k i ,i∈{0,1,2} represents the source coding length of each link, k i Different settings will result in two strategies; calculate the interruption probability under different strategies:

[0046] (1) When the link source coding length k i >n(1-H b (D) Independent decoding of the base station to user link or the base station to UAV relay to user link, the interruption probability in this case is denoted as The interruption probability of this strategy is calculated as follows:

[0047]

[0048] In the formula, ∈ i (k i The formula for calculating the frame error rate under finite-length coding is as follows:

[0049]

[0050] In the formula, Represents the integral of the Gaussian distribution The function, where C represents channel capacity, n represents coding length, and V represents channel dispersion, which is defined in this scenario model as:

[0051]

[0052] In the formula, γ represents the instantaneous signal-to-noise ratio of the link; e represents the base of the natural logarithm;

[0053] (2) When the link source coding length k i ≤n(1-H b In case (D), independent decoding is insufficient to meet information quality requirements, necessitating joint decoding to achieve communication. The interruption probability in this scenario is denoted as... The calculation process is as follows:

[0054]

[0055] Under this strategy, k i The settings are constrained. First, the selection of k0 and k2 should satisfy the precondition that frame errors occur in both independent links. After setting k0 and k2, the selection constraints for k1 are as follows:

[0056]

[0057] In the formula

[0058]

[0059] Among them, R S1 H represents the coding rate of link 1. b (.) represents binary entropy;

[0060] Choose the minimum value that satisfies the above formula as k1;

[0061] Steps 2-3: The optimization objective is to minimize the average user interruption probability under different strategies by optimizing the drone trajectory;

[0062] The optimization problem is expressed as:

[0063]

[0064] q[1]=q I ,,0000000

[0065] 0000||q[t+1]-q[t]||≤v max .

[0066] In the formula, (x q [t],y q [t]) represents the coordinate position of the UAV relay at time t, [x min ,xmax ] and [y min ,y max [] indicates the service area boundary of the drone relay throughout the entire communication process; K represents the total number of users.

[0067] Furthermore, step 3 specifically includes:

[0068] The optimization of UAV relay communication trajectory under finite-length coding is transformed into a Markov decision process;

[0069] An MDP consists of the following parts:

[0070] (1) State space The definition is as follows:

[0071] These are the coordinates of the drone's entire horizontal position.

[0072] (2) State Space The definition is as follows:

[0073] For the single-step displacement of the UAV in the horizontal direction, by limiting (Δx) q [t],Δy q [t]) to limit the flight speed and direction of the drone;

[0074] (3) Reward function:

[0075] The reward function is set based on the interruption probability calculation formula derived in step 2, and is defined as follows:

[0076]

[0077] In the formula, R s [t]、R J [t] represents the reward function at time t under the independent decoding and joint decoding strategies, respectively, R p This indicates a penalty, meaning that if the drone flies out of the environmentally restricted area, it will receive a negative penalty; specifically, under different communication strategies, it is defined as follows:

[0078]

[0079] Where, μ J ,μ S This is the proportionality coefficient;

[0080] (4) State transition probability matrix:

[0081] State transition probability matrix Using a model-free approach, the agent, i.e., the drone, can learn and optimize its strategy based on historical experience data. The agent does not rely on the state transition probability matrix when performing tasks, so there is no need to pre-build this information.

[0082] Furthermore, step 4 specifically includes:

[0083] Step 4-1: Construct a joint optimization algorithm network for UAV mobility and communication based on the DDPG algorithm framework;

[0084] The DDPG-based joint optimization algorithm for UAV mobility and communication comprises four networks: an actor policy network with parameter θ, a critic evaluation network with parameter ω, and a network with parameter ω. The actor target policy network and a parameter of The target evaluation network is a critic network; wherein the trainable network is an actor policy network with parameter θ and a critic evaluation network with parameter ω, and the target network has the same network structure as its corresponding trainable network.

[0085] actor policy network μ θ This network estimates the communication and flight strategies of the UAV. The actor policy network generates deterministic actions. Therefore, this network takes a state vector as input and outputs a deterministic action decision. The target actor policy network has the same structure as the actor policy network. The target actor policy network takes the next state as input and outputs the optimal action in the next state.

[0086] critic's review of the online Q ω The network evaluates the action value function of the policy learned by the actor policy network, thereby guiding the actor policy network to update; the critic evaluation network takes the state vector and action as input and outputs the Q-value of the state-action pair; the target critic evaluation network has the same structure as the critic evaluation network. The input of the target critic evaluation network is the next state and the optimal action output by the target actor policy network in the next state. The output is the Q-value of the input state-action pair, which is used to calculate the target Q-value.

[0087] Experience pool: Uses an experience replay pool with a capacity of E. Store historical sample data; during training, randomly select a batch of samples of size G from the experience pool, where G << E, and input them into each network to update the network parameters.

[0088] Step 4-2: Draw a batch of samples (s) from the experience pool. n ,a n ,rn ,s n+1 ), update the network; s n ,a n ,r n ,s n+1 These represent the current state, the action taken in that state, the immediate reward received, and the next state transitioned to after performing the action, respectively.

[0089] actor policy network μ θ First, extract s from the sample n Input the current actor policy network μ θ To obtain the continuous actions output by the network μ(.) represents the current actor policy network; then s n , Input critic evaluation network Q ω The state-action value function is obtained and used as the loss function to optimize and update the network parameters θ; the loss function is expressed as:

[0090]

[0091] Where Q(.) represents the evaluation network, and G represents the number of samples drawn from the experience pool. Indicates action gradient, This represents the gradient operation with respect to the policy network parameters θ;

[0092] critic's review of the online Q ω First, the target value for network updates is calculated, which is the value of s extracted from the samples. n+1 Input actor target policy network The target action is obtained from the output. Then s n+1 , Input critic target evaluation network In the process, the value function is obtained; the obtained value function is then compared with r in the extracted sample. n The objective function is constructed as follows:

[0093]

[0094] In the formula, η represents the attenuation factor; r n This indicates that in state s n Next, execute action a n The instant reward obtained afterward;

[0095] The loss function is constructed as follows:

[0096]

[0097] in, This represents the gradient of the parameter ω of the critic evaluation network;

[0098] Target Network: Adjusting the actor target policy network using a soft update strategy. and critic target evaluation network The update method is as follows:

[0099]

[0100] Where τ represents the weight coefficient of soft update;

[0101] Step 4-3: Overall Process;

[0102] Step 4-3-1: Perform a process on the actor policy network μ θ Critic rating network Q ω and actor target strategy network critic target evaluation network The parameters θ, ω, as well as Perform initialization, simultaneously initializing the communication system environment and the experience pool.

[0103] Step 4-3-2: At each time t, the drone, acting as an intelligent agent, observes the state s of the environment. t Subsequently, the actor policy network μ θ Obtain action decision a t The drone will randomly explore the environment until it learns a superior strategy. In this algorithm, the drone's exploration of the environment is manifested by superimposing a Gaussian-distributed noise with a variance that gradually decreases over time onto continuous actions.

[0104]

[0105] in, Indicates the exploration noise term;

[0106] Step 4-3-3: After obtaining the next action, the drone will execute action a. t And calculate the corresponding reward r t At the same time, the environment will transition to the next state s. t+1 Meanwhile, state s t Action a t Rewards r t And the next state s t+1 This will constitute the sample quadruple (s) t ,a t ,r t,s t+1 Store in the experience pool This information is used for subsequent network training.

[0107] Step 4-3-4: Update and train the network to improve the overall performance of the system; when the number of data samples in the experience pool reaches a certain amount, the network will be triggered to start training. At this time, the algorithm will randomly draw a small batch of samples from the experience pool and then train the actor policy network μ. θ and the critic rating network Q ω Update; then update the actor target policy network. critic target evaluation network Update.

[0108] An electronic device includes: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the above-described trajectory optimization method.

[0109] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the above-described trajectory optimization method.

[0110] A chip includes a processor for retrieving and running a computer program from a memory, causing a device on which the chip is mounted to perform the trajectory optimization method described above.

[0111] A computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the above-described trajectory optimization method.

[0112] The beneficial effects of this invention are as follows:

[0113] This invention constructs a trajectory optimization model for a lossy relay UAV communication system under finite-length coding, based on finite-length coding theory and reinforcement learning algorithms. On this basis, the performance limit of the UAV trajectory under optimal conditions is derived, enabling low-complexity optimization of the UAV trajectory and reducing the probability of system outages. The trajectory optimization method of this invention can support the application of UAVs in future 6G communication systems, particularly in emergency communication and remote area coverage scenarios, where it has significant application value. Attached Figure Description

[0114] Figure 1 This is a diagram of a lossy cooperative UAV relay communication model under finite-length coding.

[0115] Figure 2This is a schematic diagram of independent decoding of the link from the base station to the drone relay to the user.

[0116] Figure 3 This is a schematic diagram of independent decoding of the base station to user link.

[0117] Figure 4 This is a flowchart of a joint optimization algorithm for UAV mobility and communication based on the DDPG algorithm.

[0118] Figure 5 This is a graph showing the change in interruption probability when k0 = 80 in an embodiment of the present invention.

[0119] Figure 6 This is a graph showing the change in interruption probability when k0 = 100 in an embodiment of the present invention.

[0120] Figure 7 This is a graph showing the change in interruption probability when k0 = 120 and 140 in an embodiment of the present invention.

[0121] Figure 8 This is under strategy 1 of the present invention, different k i The image shows the trajectory of the drone.

[0122] Figure 9 This is under strategy 2 of the present invention, different k i The curve showing the change in the interruption probability under the given value.

[0123] Figure 10 This is under strategy 2 of the present invention, different k i The image shows the trajectory of the drone.

[0124] Figure 11 This is a performance comparison chart of different algorithm frameworks in embodiments of the present invention.

[0125] Figure 12 This is a graph showing the impact of different discount coefficient η values ​​on algorithm performance in an embodiment of the present invention.

[0126] Figure 13 This is a graph showing the impact of different exploration noise values ​​on algorithm performance in embodiments of the present invention. Detailed Implementation

[0127] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0128] This invention provides a method for optimizing the trajectory of a relay UAV. First, a lossy relay UAV communication system model under finite-length coding is constructed, encompassing communication time slot partitioning, UAV flight constraints, and channel models. Next, the outage probability expressions for two communication strategies under finite-length coding are derived: independent decoding and joint decoding strategies. Then, the trajectory optimization problem is transformed into a Markov Decision Process (MDP), clearly defining the state space, action space, reward function, and state transition mechanism. Subsequently, a joint optimization algorithm based on Deep Deterministic Policy Gradient (DDPG) is used to solve the problem, achieving joint optimization of the UAV trajectory and communication strategy. Finally, the optimal UAV trajectory and strategy configuration are output, providing a reference for practical system deployment. This invention can effectively reduce the system outage probability and improve communication reliability under finite-length coding conditions, and is applicable to various scenarios in 6G and future air-to-ground cooperative communication networks.

[0129] The technical solution adopted by this invention to solve its technical problem is as follows:

[0130] Step 1: Construct a lossy relay UAV communication system model under finite-length coding, including communication time slot partitioning, UAV flight constraints, and channel model;

[0131] Step 1-1: Construct a model of a lossy cooperative UAV relay communication system under a finite encoding length scenario;

[0132] The drone is responsible for transmitting information to K>1 users on the ground, where the user set is represented as follows: This invention assumes that the communication process occurs within a finite time interval. Where T>0. The flight time of the UAV. Divide the time slots into N>1 slots, which can be represented as follows: Each time slot has a short duration. Throughout the communication process, the served users utilize frequency division multiple access (FDMA) for multi-user access. Furthermore, R... s0 R s1 R s2 These represent the coding rates of the base station to UAV link, base station to user link, and UAV to user link, respectively, and k0, k1, and k2 represent the information sequence length of each link.

[0133] In three-dimensional coordinates, the position of ground user k at time slot t can be represented as: The horizontal coordinate of the UAV is represented as q[t] = (x q [t],y q [t],z q In practical scenarios, UAVs are typically deployed at the lowest possible altitude to achieve optimal communication relay performance. Therefore, a fixed UAV altitude, i.e., z, is considered. q It is a constant. The initial position of the UAV is q. I =(xI ,y I ,z q In the communication model, the behavior of drones is constrained by the following conditions:

[0134] q[1]=q I

[0135] ||q[t+1]-q[t]||≤v max

[0136] In the formula, v max The maximum flight speed of the drone in the horizontal direction is represented by Δq[t] = (Δx) which limits the change in the drone's coordinates. q [t],Δy q To achieve this, use [t],0);

[0137] Step 1-2: Construct a lossy relay UAV communication channel model under finite-length coding;

[0138] For the A2G channel, it is further assumed that the A2G channel conditions depend on the UAV's elevation angle and environmental influences, including the Loss of Position (LoS). A probabilistic line-of-sight (LOS) channel model based on elevation angle is used for channel modeling. In the communication link, the line-of-sight link probabilities at the transmitter and receiver are... Non-line-of-sight link probability It can be represented as:

[0139]

[0140] In the formula, These are environmental parameters, affected by environmental conditions, and should be set according to the actual communication scenario. θ t,r The elevation angle between the transmitting and receiving ends of the link can be expressed as:

[0141]

[0142] In the formula, ΔH represents the height difference between the transmitting and receiving ends of the link, and d t,r The distance between the sending and receiving ends of the link can be calculated using the following formula:

[0143]

[0144] At this point, the path loss for line-of-sight links and non-line-of-sight links can be expressed as follows:

[0145]

[0146] In the formula, f represents the carrier frequency, and η Los η NLoSThis represents the additional path loss. Using the average path loss as the path loss between the receiver and transmitter of each link, the average path loss PL between the drone and the k-th user is... k Represented as:

[0147]

[0148] Assume the given base station transmit power is P b Given the UAV's transmit power P UAV Then the signal power received by the k-th user from the base station and from the drone and They are represented as follows:

[0149]

[0150] In the formula, |g| 2 Indicates channel fading gain;

[0151] Furthermore, the instantaneous signal-to-noise ratio of each link between the k-th user and the base station and the drone can be expressed as:

[0152]

[0153]

[0154] In the formula, N0 represents the noise power spectral density of additive Gaussian noise.

[0155] Step 2: Derive the interruption probability expressions for two communication strategies under finite-length coding, namely independent decoding and joint decoding strategies;

[0156] Step 2-1: For the system model established in Step 1, consider three scenarios where communication will not be interrupted;

[0157] (1) Considering that the base station to user link is independently decoded and no frame errors occur, the interruption probability under this case is denoted as P1;

[0158] (2) Considering the case where frame errors occur in the base station to user link, the base station to UAV link and the UAV to user link are decoded independently and no frame errors occur. The interruption probability in this case is denoted as P2.

[0159] (3) Considering that the base station to UAV link is decoded independently and no frame errors occur, the base station to user and UAV to user links are decoded jointly to achieve no frame errors. The interruption probability in this case is denoted as P3.

[0160] Step 2-2: Assume k i ,i∈{0,1,2} represents the source coding length of each link, k iDifferent settings will result in two strategies; calculate the interruption probability under different strategies:

[0161] (1) When the link source coding length k i With a larger setting, the base station to user link or the base station to drone relay to user link is decoded independently. The probability of interruption in this case is denoted as... The interruption probability of this strategy is calculated as follows:

[0162]

[0163] In the formula, ∈ i (k i The formula for calculating the frame error rate under finite-length coding is as follows:

[0164]

[0165] In the formula, Represents the integral of the Gaussian distribution The function, where V represents channel dispersion, is defined in this scenario model as:

[0166]

[0167] In the formula, γ represents the instantaneous signal-to-noise ratio of the link;

[0168] (2) When the link source coding length k i When the setting is small, independent decoding is insufficient to meet the information quality requirements, necessitating joint decoding to achieve communication. The interruption probability in this case is denoted as... The calculation process is as follows:

[0169]

[0170] Under this strategy, k i The settings are constrained. First, the selection of k0 and k2 should satisfy the precondition that frame errors occur in both independent links. After setting k0 and k2, it is easy to see that the constraint on the selection of k1 is expressed as follows:

[0171]

[0172] In the formula

[0173]

[0174] Choose the minimum value that satisfies the above formula as k1;

[0175] Steps 2-3: The optimization objective is to minimize the average user interruption probability under different strategies by optimizing the drone trajectory;

[0176] The optimization problem can be expressed as:

[0177]

[0178] q[1]=q I ,,0000000

[0179] 0000||q[t+1]-q[t]||≤v max .

[0180] In the formula, (x q [t],y q [t]) represents the coordinate position of the UAV relay at time t, [x min ,x max ] and [y min ,y max [] indicates the service area boundary range of the drone relay throughout the entire communication process.

[0181] Step 3: Transform the trajectory optimization problem into a Markov Decision Process (MDP), and define the state space, action space, reward function, and state transition mechanism;

[0182] The optimization of UAV relay communication trajectory under finite-length coding is transformed into a Markov decision process;

[0183] An MDP consists of the following parts:

[0184] (1) State space The definition is as follows:

[0185] These are the coordinates of the drone's entire horizontal position.

[0186] (2) State space The definition is as follows:

[0187] For the single-step displacement of the UAV in the horizontal direction, by limiting (Δx) q [t],Δy q [t]) to limit the drone's flight speed and direction;

[0188] (3) Reward function:

[0189] The reward function is set based on the interruption probability calculation formula derived in step 2, and is defined as follows:

[0190]

[0191] In the formula, R s [t]、R J [t] represents the reward function at time t under the independent decoding and joint decoding strategies, respectively, Rp This represents a penalty, meaning that if the drone flies outside the environmentally restricted area, it will receive a negative penalty. Specifically, under different communication strategies, it is defined as follows:

[0192]

[0193] Where, μ J ,μ s This is a scaling factor, which is adjusted according to experimental needs to scale the reward value and accelerate algorithm convergence.

[0194] (4) State transition probability matrix:

[0195] Typically, the state transition probability matrix This invention relates to wireless communication environments, and the algorithm employs a model-free approach, enabling the agent (i.e., a drone) to learn and optimize its strategy based on historical experience data. This means that the agent does not rely on the state transition probability matrix when performing tasks, thus eliminating the need for pre-construction of this information.

[0196] Step 4: Use a joint optimization algorithm based on Deep Deterministic Policy Gradient (DDPG) to solve the problem and achieve joint optimization of UAV trajectory and communication strategy;

[0197] Step 4-1: Construct a joint optimization algorithm network for UAV mobility and communication based on the DDPG algorithm framework;

[0198] The DDPG-based joint optimization algorithm for UAV mobility and communication comprises four networks: an actor policy network with parameter θ, a critic evaluation network with parameter ω, and a corresponding network with parameter ω. The actor target policy network and a parameter of The target evaluation network consists of an actor policy network with parameter θ and a critic evaluation network with parameter ω. The target network has the same network structure as its corresponding trainable network.

[0199] actor policy network μ θ This network estimates the communication and flight strategies of the UAV. In the DDPG-based joint optimization algorithm for UAV mobility and communication, the actor policy network will generate deterministic actions. Therefore, the network takes a state vector as input and outputs a deterministic action decision. The target actor policy network has the same structure as the actor policy network. Similarly, the target actor policy network takes the next state as input and outputs the optimal action in the next state.

[0200] critic's review of the online Qω This network evaluates the action-value function of the policy learned by the actor policy network, thereby guiding the actor network to update and bringing its learned policy closer to the optimal policy, which is more conducive to maximizing the reward of the communication system. Therefore, the critic evaluation network takes a state vector and actions as input and outputs the Q-value of the state-action pair. The target critic evaluation network has the same structure as the critic evaluation network. Its input is the next state and the optimal action output by the target actor policy network in the next state. Its output is the Q-value of the input state-action pair, which is used to calculate the target Q-value.

[0201] Experience pool: Uses an experience replay pool with a capacity of E. This is used to store historical sample data. During training, a small batch of samples (of size G, where G << E) is randomly drawn from the experience pool and input into each network to update the network parameters;

[0202] Step 4-2: Draw a batch of samples (s) from the experience pool. n ,a n ,r n ,s n+1 Update the network;

[0203] actor policy network μ θ First, extract s from the sample n Input the current actor policy network μ θ To obtain the continuous actions output by the network Then s n , Input critic evaluation network Q ω The state-action value function is obtained and used as the loss function to optimize and update the network parameters θ. The loss function is expressed as:

[0204]

[0205] critic's review of the online Q ω The update of this network is relatively complex. First, the target value for the network update needs to be calculated, which is to extract the s value from the samples. n+1 Input actor target policy network The target action is obtained from the output. Then s n+1 , Input critic target evaluation network In the process, the value function is obtained. The obtained value function is then compared with r from the extracted samples. n The objective function is constructed as follows:

[0206]

[0207] In the formula, η represents the decay factor, which determines the weight of the value function of the next action and needs to be adjusted to achieve a suitable prediction result.

[0208] The objective function described above will be further used to construct the loss function as follows:

[0209]

[0210] Target Network: In the reinforcement learning framework of this invention, a soft update strategy is used to adjust the actor target policy network. and critic target evaluation network The update method is as follows:

[0211]

[0212] Step 4-3: The overall process of executing the algorithm;

[0213] First, the algorithm applies the actor policy network μ θ Critic rating network Q ω and actor target strategy network critic target evaluation network The parameters θ, ω, as well as Perform initialization, and simultaneously initialize the communication system environment (e.g., the initial position of the UAV) and the experience pool.

[0214] At each time t, the drone, acting as an intelligent agent, observes the state s of the environment. t Subsequently, the actor policy network μ θ Obtain action decision a t Similarly, the drone will first randomly explore the environment until it learns a superior strategy. In this algorithm, the drone's exploration of the environment is represented by superimposing a Gaussian-distributed noise with a variance that gradually decreases over time onto continuous actions, i.e.:

[0215]

[0216] The algorithm will keep the control actions within a reasonable range, and after obtaining the next action, the drone will execute action a. t And calculate the corresponding reward r t At the same time, the environment will transition to the next state s. t+1 At the same time, state s t Action a t Rewards r t And the next state st+1 This will constitute the sample quadruple (s) t ,a t ,r t ,s t+1 Store in the experience pool This information is used for subsequent network training.

[0217] Finally, the network is updated and trained to improve the overall performance of the system. When the number of data samples in the experience pool reaches a certain amount, the network will be triggered to start training. At this time, the algorithm will randomly draw a small batch of samples from the experience pool and then train the actor policy network μ. θ and the critic rating network Q ω Update the network. Then, update the actor target policy network. critic target evaluation network Update slowly.

[0218] Step 5: Output the optimal UAV trajectory and strategy configuration to provide a reference for actual system deployment.

[0219] Example:

[0220] In the simulation analysis of the embodiments of this invention, a lossy cooperative drone relay communication network consisting of one drone and K=2 users was constructed. This communication scenario was further divided into three cases: independent decoding of the base station-to-user link without frame errors; independent decoding of both the base station-to-drone and drone-to-user links without frame errors, even when frame errors occur in the base station-to-user link; and independent decoding of the base station-to-drone link without frame errors, combined decoding of the base station-to-user and drone-to-user links without frame errors. These three cases are summarized as two communication strategies: independent decoding and combined decoding. The network simulation environment was set within a 20×20km area. The drone's flight altitude was set to 500 meters, and its initial position was the same as the base station's position.

[0221] In the DDPG-based UAV mobility and communication joint optimization algorithm proposed in this invention, all actor policy networks contain three fully connected layers, and the critic evaluation network contains four fully connected layers. Each fully connected layer consists of 128 neurons. For the actor policy network, the first and second layers use the ReLU activation function, and the output layer uses the tanh activation function to scale the output, ensuring the action values ​​are within a reasonable range. For the critic evaluation network, each fully connected layer uses the ReLU activation function. The algorithm is trained for 300 rounds, with a maximum of 200 training steps per round. The simulations involved in this invention were performed using Python 3.8 and PyTorch 1.12.1.

[0222] (1) Simulation results of Strategy 1;

[0223] Through a series of simulation experiments, three sets of representative data charts were obtained, which intuitively demonstrate k. i The impact of value changes on system performance. First, from... Figure 5 It can be observed that when k0 is fixed at 80, and k1 and k2 are taken as 100, 120, 140, and 160 respectively, the interruption probability shows a significant decreasing trend. This indicates that as k1 and k2 increase, the user interruption probability decreases, the system robustness is enhanced, and the risk of communication interruption is effectively reduced.

[0224] Figure 6 This trend was further verified. Here, k0 was set to 100, while the combinations of k1 and k2 were 100 and 120, 120 and 140, and 120 and 160, respectively. Simulation results consistently showed that as the value of k increased, the interruption probability continuously decreased, especially when k1 and k2 were high, resulting in a more significant improvement in system performance.

[0225] Figure 7 This shows the cases where k0 is 120 and 140, with combinations of k1 and k2 of 120 and 140, 120 and 160, and 140 and 160. These data again emphasize the positive effect of increasing the k value on reducing the probability of interruption. Especially when k0, k1, and k2 all take high values, the system exhibits better stability and reliability. This is because, under the same system bandwidth, k... i Increasing the value of k means a higher coding rate and more useful information carried per information unit. Combining the three simulation results, we can conclude that under the independent decoding framework of Strategy 1, appropriately increasing the value of k can effectively improve system stability and reduce the probability of interruption. However, in reality, due to limitations such as bandwidth and signal-to-noise ratio (SNR), a high coding rate may require a higher transmission rate, which could lead to a decrease in SNR and thus an increase in the bit error rate. Therefore, the optimal value of k varies depending on the application scenario. i The choice of value warrants further study.

[0226] Figure 8 Describes different k strategies for strategy 1 i The final trajectory of the UAV is obtained by training based on the algorithm proposed in this invention. It can be observed that regardless of k... i Regardless of the value of ki, the algorithm can ensure that the trained drone reaches the target position with the minimum interruption probability through a reasonable flight trajectory. Furthermore, the target position is not affected by the value of ki, and the drone ultimately stays near the optimal position.

[0227] (2) Simulation results of Strategy 2;

[0228] Figure 9For different parameters k i The impact of k0 and k2 values ​​on the system interruption probability under joint decoding of strategy 2 is investigated. It can be concluded that the interruption probability decreases rapidly with increasing iteration count. This decrease is particularly significant when both k0 and k2 take high values ​​(e.g., 80, 80). After more than 100 iterations, all k... i The interruption probabilities of value combinations tend to stabilize and remain at a low level. Compared to the independent decoding case in Strategy 1, at a lower k... i Even under certain conditions, it can still achieve good system performance. Figure 10 Describes different k strategies for strategy 2 i The value is the final trajectory of the UAV obtained by training based on the algorithm proposed in this invention. It can also be observed that the target position is not affected by k. i The influence of the value of k, regardless of i Regardless of the value, the drone will always reach the same optimal interruption probability position.

[0229] (3) Algorithm performance comparison and analysis;

[0230] Figure 11 The performance of the algorithm proposed in this invention was compared with that of other algorithmic frameworks through simulation. Taking strategy 1, k0 = k1 = 100, k2 = 120 as an example, it can be seen that the algorithm proposed in this invention is comparable to the TD3 algorithm in terms of convergence speed and final reward value. However, the UAV mobility and communication joint optimization algorithm based on DDPG proposed in this invention shows significantly better stability after convergence than the TD3 algorithm. This is because the TD3 algorithm has relatively high algorithmic complexity, introducing more mechanisms than DDPG, such as dual-Q network, target policy smoothing, and delayed policy update; and the TD3 algorithm has low parameter sensitivity, making parameter tuning very complex. The UAV mobility and communication joint optimization algorithm based on DDPG proposed in this invention has undergone fine parameter tuning to achieve optimal performance, thus resulting in slightly better performance.

[0231] Figure 12 , Figure 13 The impact of different parameters on algorithm performance, among which, Figure 12 Simulation results for different discount coefficients η, Figure 13 The impact of different exploration noise values ​​on algorithm performance.

[0232] In the DDPG algorithm, the commonly used value range for the discount factor η is typically between 0.9 and 0.99. This invention conducted experiments on different discount factors η, and simulation results show that the value of the discount factor η affects the algorithm's performance. A reasonable selection of the value can avoid the problems of "myopia" (over-focusing on immediate rewards) and "hyperopia" (over-focusing on long-term rewards while ignoring recent rewards) that occur during the learning process. This is because a larger discount factor η means that the agent will place greater emphasis on future rewards, which is helpful for learning long-term strategies. The specific value needs to be adjusted according to the characteristics of the task and the experimental results. Figure 12 Analysis and comparison show that η = 0.95 is a suitable value, at which point the algorithm performs better.

[0233] The policy network in the algorithm outputs deterministic actions, therefore, noise needs to be superimposed on the actions to allow the agent to explore the environment randomly, which helps in discovering better policies. Excessive noise variance may prevent the agent from learning effectively, while too small variance may lead to insufficient exploration. Simulation results... Figure 13 As can be seen, both large and small noise factors degrade algorithm performance. When the noise factor is 3, the algorithm achieves good convergence. In practical applications, the noise variance needs to be adjusted according to the specific task to obtain appropriate exploration results and better algorithm performance.

Claims

1. A method for optimizing the trajectory of a relay UAV, characterized in that, Includes the following steps: Step 1: Construct a lossy relay UAV communication system model under finite-length coding, including communication time slot partitioning, UAV flight constraints, and channel model; Step 2: Derive the interruption probability expressions for two communication strategies under finite-length coding, namely independent decoding and joint decoding strategies; Step 3: Transform the trajectory optimization problem into a Markov Decision Process (MDP), and define the state space, action space, reward function, and state transition mechanism; Step 4: Solve the problem using a joint optimization algorithm based on Deep Deterministic Policy Gradient (DDPG) to achieve joint optimization of UAV trajectory and communication strategy; Step 5: Output the optimal UAV trajectory and strategy configuration to provide a reference for actual system deployment.

2. The method for optimizing the trajectory of a relay UAV according to claim 1, characterized in that, Step 1 specifically includes: Step 1-1: Construct a model of a lossy cooperative UAV relay communication system under a finite encoding length scenario; The drone is responsible for transmitting information to K>1 users on the ground, where the user set is represented as follows: Assume the communication process occurs within a finite time interval. Where T>0; the flight time T of the UAV is divided into N>1 time slots, and the set of time slots is represented as follows. t represents the time slot; during the communication process, the served users use frequency division multiple access for multi-user access; Let R s0 R s1 R s2 These represent the coding rates of the base station to UAV link, base station to user link, and UAV to user link, respectively, and k0, k1, and k2 represent the information sequence length of each link. In three-dimensional coordinates, the position of ground user k at time slot t is represented as follows: The coordinates of the UAV are represented as q[t] = (x q [t],y q [t],z q ), z q It is a constant. Let x and y represent the coordinates of ground user k at time slot t, respectively. q [t],y q [t],z q Let x, y, and z represent the coordinates of the UAV at time slot t, respectively, and let q be the initial position of the UAV. I =(x I ,y I ,z q In the communication model, the behavior of drones is constrained by the following conditions: q[1]=q I ||q[t+1]-q[t]||≤v max In the formula, v max The maximum flight speed of the drone in the horizontal direction is represented by Δq[t] = (Δx) which limits the change in the drone's coordinates. q [t],Δy q [t],0) is used to achieve this; q[t+1] represents the coordinate representation of the UAV in time slot t+1; Step 1-2: Construct a lossy relay UAV communication channel model under finite-length coding; Channel modeling is performed using a probabilistic line-of-sight channel model based on elevation angle. In the communication link, the line-of-sight link probabilities at the transmitting and receiving ends are... Non-line-of-sight link probability Represented as: In the formula, For environmental parameters, θ t,r The elevation angle between the transmitting and receiving ends of the link is expressed as: In the formula, ΔH represents the height difference between the transmitting and receiving ends of the link, and d t,r The distance between the sending and receiving ends of the link can be calculated using the following formula: Where, x t y t z t These represent the x-axis, y-axis, and z-axis coordinates of the transmitting end, respectively. r y r z r These represent the x-axis, y-axis, and z-axis coordinates of the receiving end, respectively. At this point, the path losses for line-of-sight links and non-line-of-sight links are expressed as follows: In the formula, f represents the carrier frequency, and η Los η NLoS Indicates additional path loss; η Los Representing a line-of-sight scenario, η NLoS Indicates a non-line-of-sight scene; Using the average path loss as the path loss between the receiver and transmitter of each link, the average path loss PL between the UAV and the k-th user is... k Represented as: Assume the given base station transmit power is P b Given the UAV's transmit power P UAV Then the signal power received by the k-th user from the base station and from the drone and They are represented as follows: In the formula, |g| 2 Indicates channel fading gain; The instantaneous signal-to-noise ratio of each link between the k-th user and the base station and the drone is expressed as: In the formula, N0 represents the noise power spectral density of additive Gaussian noise; γ b,k γ represents the instantaneous signal-to-noise ratio of the link between the k-th user and the base station. UAV,k This represents the instantaneous signal-to-noise ratio of the link between the k-th user and the drone.

3. The method for optimizing the trajectory of a relay UAV according to claim 2, characterized in that, Step 2 specifically includes: Step 2-1: For the lossy cooperative UAV relay communication system model established in Step 1, consider three scenarios in which communication will not be interrupted; (1) Considering that the base station to user link is independently decoded and no frame errors occur, the interruption probability under this case is denoted as P1; (2) Considering the case where frame errors occur in the base station to user link, the base station to UAV link and the UAV to user link are decoded independently and no frame errors occur. The interruption probability in this case is denoted as P2. (3) Considering that the base station to UAV link is decoded independently and no frame errors occur, the base station to user and UAV to user links are decoded jointly to achieve no frame errors. The interruption probability in this case is denoted as P3. Step 2-2: Assume k i ,i∈{0,1,2} represents the source coding length of each link, k i Different settings will result in two strategies; calculate the interruption probability under different strategies: (1) When the link source coding length k i >n(1-H b (D) Independent decoding of the base station to user link or the base station to UAV relay to user link, the interruption probability in this case is denoted as The interruption probability of this strategy is calculated as follows: In the formula, ∈ i (k i The formula for calculating the frame error rate under finite-length coding is as follows: In the formula, Represents the integral of the Gaussian distribution The function, where C represents channel capacity, n represents coding length, and V represents channel dispersion, which is defined in this scenario model as: In the formula, γ represents the instantaneous signal-to-noise ratio of the link; e represents the base of the natural logarithm; (2) When the link source coding length k i ≤n(1-H b In case (D), independent decoding is insufficient to meet information quality requirements, necessitating joint decoding to achieve communication. The interruption probability in this scenario is denoted as... The calculation process is as follows: Under this strategy, k i The settings are constrained. First, the selection of k0 and k2 should satisfy the precondition that frame errors occur in both independent links. After setting k0 and k2, the selection constraints for k1 are as follows: In the formula Among them, R S1 H represents the coding rate of link 1. b (.) represents binary entropy; Choose the minimum value that satisfies the above formula as k1; Steps 2-3: The optimization objective is to minimize the average user interruption probability under different strategies by optimizing the drone trajectory; The optimization problem is expressed as: In the formula, (x q [t],y q [t]) represents the coordinate position of the UAV relay at time t, [x min ,x max ] and [y min ,y max [] indicates the service area boundary of the drone relay throughout the entire communication process; K represents the total number of users.

4. The method for optimizing the trajectory of a relay UAV according to claim 3, characterized in that, Step 3 specifically includes: The optimization of UAV relay communication trajectory under finite-length coding is transformed into a Markov decision process; An MDP consists of the following parts: (1) State space The definition is as follows: These are the coordinates of the drone's entire horizontal position. (2) State Space The definition is as follows: For the single-step displacement of the UAV in the horizontal direction, by limiting (Δx) q [t],Δy q [t]) to limit the drone's flight speed and direction; (3) Reward function: The reward function is set based on the interruption probability calculation formula derived in step 2, and is defined as follows: In the formula, R s [t]、R J [t] represents the reward function at time t under the independent decoding and joint decoding strategies, respectively, R p This indicates a penalty, meaning that if the drone flies out of the environmentally restricted area, it will receive a negative penalty; specifically, under different communication strategies, it is defined as follows: Where, μ J ,μ S This is the proportionality coefficient; (4) State transition probability matrix: State transition probability matrix Using a model-free approach, the agent, i.e., the drone, can learn and optimize its strategy based on historical experience data. The agent does not rely on the state transition probability matrix when performing tasks, so there is no need to pre-build this information.

5. The method for optimizing the trajectory of a relay UAV according to claim 4, characterized in that, Step 4 specifically includes: Step 4-1: Construct a joint optimization algorithm network for UAV mobility and communication based on the DDPG algorithm framework; The DDPG-based joint optimization algorithm for UAV mobility and communication comprises four networks: an actor policy network with parameter θ, a critic evaluation network with parameter ω, and a network with parameter ω. The actor target policy network and a parameter of The target evaluation network is a critic network; wherein the trainable network is an actor policy network with parameter θ and a critic evaluation network with parameter ω, and the target network has the same network structure as its corresponding trainable network. actor policy network μ θ This network estimates the communication and flight strategies of the UAV. The actor policy network generates deterministic actions. Therefore, this network takes a state vector as input and outputs a deterministic action decision. The target actor policy network has the same structure as the actor policy network. The target actor policy network takes the next state as input and outputs the optimal action in the next state. critic's review of the online Q ω The network evaluates the action value function of the policy learned by the actor policy network, thereby guiding the actor policy network to update; the critic evaluation network takes the state vector and action as input and outputs the Q-value of the state-action pair; the target critic evaluation network has the same structure as the critic evaluation network. The input of the target critic evaluation network is the next state and the optimal action output by the target actor policy network in the next state. The output is the Q-value of the input state-action pair, which is used to calculate the target Q-value. Experience pool: An experience replay pool with a capacity of E is adopted to store historical sample data; during the training process, a batch of samples with a size of G (where G << E) is randomly drawn from the experience pool and input into each network to update the network parameters; Step 4-2: Draw a batch of samples (s) from the experience pool. n ,a n ,r n ,s n+1 ), update the network; s n ,a n ,r n ,s n+1 These represent the current state, the action taken in that state, the immediate reward received, and the next state transitioned to after performing the action, respectively. actor policy network μ θ First, extract s from the sample n Input the current actor policy network μ θ To obtain the continuous actions output by the network μ(.) represents the current actor policy network; then s n , Input critic evaluation network Q ω The state-action value function is obtained and used as the loss function to optimize and update the network parameters θ; the loss function is expressed as: Where Q(.) represents the evaluation network, and G represents the number of samples drawn from the experience pool. Indicates action gradient, This represents the gradient operation with respect to the policy network parameters θ; critic's review of the online Q ω First, the target value for network updates is calculated, which is the value of s extracted from the samples. n+1 Input actor target policy network The target action is obtained from the output. Then s n+1 , Input critic target evaluation network In the process, the value function is obtained; the obtained value function is then compared with r in the extracted sample. n The objective function is constructed as follows: In the formula, η represents the attenuation factor; r n This indicates that in state s n Next, execute action a n The instant reward obtained afterward; The loss function is constructed as follows: in, This represents the gradient of the parameter ω of the critic evaluation network; Target Network: Adjusting the actor target policy network using a soft update strategy. and critic target evaluation network The update method is as follows: Where τ represents the weight coefficient of soft update; Step 4-3: Overall Process; Step 4-3-1: Perform a process on the actor policy network μ θ Critic rating network Q ω and actor target strategy network critic target evaluation network The parameters θ, ω, as well as Perform initialization, and simultaneously initialize the communication system environment and the experience pool D; Step 4-3-2: At each time t, the drone, acting as an intelligent agent, observes the state s of the environment. t Subsequently, the actor policy network μ θ Obtain action decision a t The drone will randomly explore the environment until it learns a superior strategy. In this algorithm, the drone's exploration of the environment is manifested by superimposing a Gaussian-distributed noise with a variance that gradually decreases over time onto continuous actions. in, Indicates the exploration noise term; Step 4-3-3: After obtaining the next action, the drone will execute action a. t And calculate the corresponding reward r t At the same time, the environment will transition to the next state s. t+1 Meanwhile, state s t Action a t Rewards r t And the next state s t+1 This will constitute the sample quadruple (s) t ,a t ,r t ,s t+1 Store in the experience pool This information is used for subsequent network training. Step 4-3-4: Update and train the network to improve the overall performance of the system; when the number of data samples in the experience pool reaches a certain amount, the network will be triggered to start training; at this time, the algorithm will randomly draw a small batch of samples from the experience pool and then train the actor policy network μ. θ and the critic rating network Q ω Update; then update the actor target policy network. critic target evaluation network Update.

6. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

8. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the method as described in any one of claims 1 to 5.

Images