Real-time trajectory planning method for UAV in maritime safety communication based on reinforcement learning

Abstract

The present invention provides a real-time planning method for the trajectory of a UAV for maritime safety communication based on reinforcement learning, and belongs to the field of communication technology. It solves the problem that traditional optimization methods cannot process dynamic ocean scenes in real time. The technical scheme is as follows: Step 1, establish a UAV-ship channel model that combines large-scale fading and small-scale fading; Step 2, obtain the signal-to-noise ratio and rate received by mobile ships and eavesdropping nodes; Step 3, obtain the optimal trajectory of the UAV by solving the maximum average confidentiality rate of the system; Step 4, use Q-learning and DDPG algorithms to optimize the trajectory; Step 5, set relevant simulation parameters; Step 6, display the real-time planning results of the UAV trajectory of the two algorithms under different take-off positions. The beneficial effects of the present invention are: the present invention plans the UAV flight trajectory in real time according to the communication needs of different ships, and effectively guarantees the real-time performance of marine communications.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a method for real-time planning of trajectories of unmanned aerial vehicles for sea area safety communication based on reinforcement learning. Background Art

[0002] At present, maritime communications are mainly satellite communications and mobile communications. However, satellite communications have large transmission delays and high costs, and most of them can only provide narrowband communications. Mobile communications have a small coverage range and cannot meet the needs of offshore communications. Compared with maritime satellites and shore-based base stations, UAV-assisted marine communications have the advantages of flexible deployment and large coverage. UAVs can also get closer to target users through trajectory design, providing good line-of-sight links, thereby significantly improving transmission rates and shortening communication delays. However, UAV's LOS broadcast transmission is easily intercepted by potential eavesdroppers. Therefore, we specifically consider a UAV-assisted marine security communication scenario, in which the UAV's flight trajectory should be close to the target user while staying away from potential eavesdroppers. Since UAVs cannot land or recharge on the sea, their flight trajectory on the ocean should be planned before takeoff. At the same time, in the actual marine environment, the initial takeoff position of the UAV and the route of the ship may be random, which will lead to a time-varying dynamic environment. Therefore, existing methods cannot solve the problem of real-time trajectory planning in UAV security communications in the sea. Summary of the invention

[0003] The purpose of the present invention is to provide a real-time planning method for UAV trajectories for sea safety communications based on reinforcement learning; it solves the problem that traditional optimization methods cannot handle dynamic ocean scenes in real time, and plans UAV flight trajectories in real time according to the communication needs of different ships, effectively ensuring the real-time nature of ocean communications.

[0004] The idea of ​​the invention is: first, considering the uniqueness of ocean propagation, new channel characteristics are brought about, such as scatterer sparsity, wave motion influence, ocean surface waveguide effect, climate influence, etc. Compared with the UAV-ground free space loss model of land scenes, the present invention proposes a UAV-ship channel modeling with combined large-scale and small-scale fading, which can better reflect the influence of wave motion and ocean scattering, etc.; secondly, compared with the existing algorithms, the reinforcement learning algorithms based on Q-learning and DDPG proposed in the present invention take the initial position of the UAV into account (the UAV take-off point is randomly distributed on the shore), and the initial position of the UAV is randomly selected before each trajectory optimization, so the two proposed algorithms can solve the dynamic trajectory planning problem of different UAV take-off points; finally, compared with the existing algorithms, the reinforcement learning algorithm based on Q-learning and DDPG proposed in the present invention can learn different ship channel information obtained by shipborne AIS, so that the UAV flight trajectory can be planned in real time according to the communication needs of different ships, effectively ensuring the real-time nature of marine communications.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is as follows: the method comprises the following steps:

[0006] Step 1: Establish a UAV-ship channel model combining large-scale fading and small-scale fading;

[0007] Step 2: Obtain the ship's channel information through the ship-borne AIS, and obtain the signal-to-noise ratio and rate received by the mobile ship and the eavesdropping node respectively;

[0008] Step 3: Obtain the optimal trajectory of the UAV by solving the maximum average confidentiality rate of the system;

[0009] Step 4: The UAV motion model based on discrete space and the UAV motion model based on continuous space use Q-learning and DDPG algorithms to optimize trajectories respectively;

[0010] Step 5: Set relevant simulation parameters;

[0011] Step 6. Display the real-time planning results of UAV trajectories of the two algorithms under different take-off positions, the real-time planning results of UAV trajectories of the two algorithms under different ship tracks, the average rewards obtained by each iteration of the two algorithms, and the average confidentiality rates obtained by different algorithms under different flight times.

[0012] In step 1, it is assumed that the flight time T of the drone S is divided into very small equal intervals d t Divided into N time slots, that is, T = Nd t; In each time slot, considering that the actual sea channel characteristics are easily affected by sea surface conditions and meteorological conditions, the channel fading model of UAV-mobile ship or eavesdropping is modeled into a representation including large-scale fading and small-scale fading:

[0013]

[0014] Among them, g sd [n] is the channel power gain from UAV S to mobile ship D, From drone S to eavesdropping node E k The channel power gain of

[0015] In the formula, represents the distance from the UAV to the ship D, Indicates UAV to eavesdrop E k distance.

[0016] In step 2, the ship on the ocean travels along a pre-planned route, and the route information is obtained through the ship-borne AIS, so it is assumed that the route of ship D is:

[0017]

[0018] Among them, V d is the maximum shipping speed of the ship, d t represents the time slot, x d [n] represents the coordinate of ship D on the x-axis at time slot n, d [n+1] represents the x-axis coordinate of ship D at time slot n+1, y d [n] represents the coordinate of ship D on the y-axis at time slot n, d [n+1] represents the coordinate of ship D on the y-axis at time slot n+1, (x d [n],y d [n]) and (x d [n+1],y d [n+1]) represent the positions of the ship at time slots n and n+1 respectively;

[0019] The signal-to-noise ratio and rate received by the mobile ship D are:

[0020]

[0021] Where: SINR represents signal-to-noise ratio, SINR d represents the signal-to-dryness ratio of ship D, R d represents the speed of ship D, P s [n] represents the transmit power of the UAV; represents the noise variance at the ship D;

[0022] The signal noise ratio and rate received by the eavesdropping node are:

[0023]

[0024] In the formula, SINR represents the signal-to-noise ratio, represents the signal-to-noise ratio of the eavesdropping node E, represents the noise variance at the eavesdropping E;

[0025] After optimizing the system, the maximum average confidentiality rate of the system is expressed as:

[0026]

[0027] In the formula, It is the maximum horizontal flight speed of the UAV.

[0028] In step 4, the sea area drone trajectory optimization based on Q-learning:

[0029] First, a Q-learning framework is proposed to solve the discrete trajectory optimization problem. The Q-learning algorithm is regarded as a Markov decision process, which is represented as a set<S,A,R,P,γ> ;

[0030] S = {s1, s2, ....} represents the location or state set of the UAV in each time slot;

[0031] Among them, the state of the nth time slot can be expressed as: S n ={x s (n),y s (n),H},(x s (n),y s (n)) represents the horizontal coordinate of the UAV, and H represents the vertical coordinate of the UAV;

[0032] A represents the action set available to the UAV. In the discrete space under the Q-learning framework, there are 9 available actions {up, down, left, right, upper left, upper right, lower left, lower right, hover};

[0033] R is the reward function of the UAV, denoted as r n =r s +r b +r f ;

[0034] where r s represents the confidentiality rate reward, r b represents the penalty for UAV flying out of the boundary, r f represents the flight time penalty;

[0035] P represents the state transition probability, and P(s′|s, a) represents the probability distribution of the UAV moving to the next state s′ after taking action a in the current state s;

[0036] γ represents the discount factor, which determines whether the UAV values ​​future rewards or current rewards;

[0037] At the beginning of each iteration, a UAV take-off point on the shore is randomly selected for training, and since mobile ships follow fixed routes, the route information is obtained through the shipborne AIS historical information;

[0038] First, the UAV will select an action a according to the exploration rate ε in the nth time slot. n If the action causes the UAV to fly out of the boundary or fail to return on time, the UAV will be affected by r b and r f punishment,

[0039] Where: r b and r f is a negative constant;

[0040] Otherwise the UAV will follow a n Move to the next state s′ and get reward r n ;

[0041] Then according to the update formula:

[0042] Q π (s n , a n )←Q π (s n , a n )+α(r n +γmaxQ π (s n+1 , a n+1 )-Q π (s n , a n )) to update the Q table;

[0043] Among them, Q π (s,a)=Ε(R n =|s n =s,a n =a), which represents the expected future reward when the UAV follows the strategy π(a|s) and takes action a in state s;

[0044] represents the total discounted reward of the UAV in the nth time slot, represents the learning rate;

[0045] According to different ship channel information, training is carried out according to the above training process to effectively solve the dynamic trajectory planning problem under different UAV take-off points and different ship channels.

[0046] In step 4, the sea area drone trajectory optimization based on DDPG:

[0047] A DDPG framework is established to solve the continuous trajectory optimization problem. At the beginning of each iteration, a UAV take-off point on the shore is randomly selected for training. Since mobile ships sail along fixed routes, the route information is obtained through the ship-borne AIS historical information;

[0048] In each time slot, first add random noise n to the Actor current network N To avoid UAV falling into local optimization, action a = π θ (s)+n N ,here

[0049] Among them: Actor current network representation: responsible for updating the parameter θ through the gradient descent method, and taking the action a according to the current state s to interact with the environment to produce the next state s′ and reward r n ;

[0050] Secondly, also considering the issue of boundaries and flight time, when the action causes the UAV to fly out of the boundary or fail to return on time, the UAV will be affected by r b and r f The penalty is, where r b and r f is a negative constant;

[0051] Otherwise the UAV will follow a n Move to the next state s′ and get reward r n ; Then the corresponding element (s n , a n , r n ,s n+1 ) into the experience pool R b middle;

[0052] When the experience pool is full, batch sampling is performed from the experience pool and the loss function is used to calculate the and the gradient function To update the parameters θ and ω of the Critic current network and the Actor current network respectively, where N b is the number of samples in the mini-batch, and y n =r+γQ′(s′,a′,ω′);

[0053] Among them: Critic current network representation: responsible for calculating the current Q(s, a, ω) value, and according to the target value y n =r+γQ′(s′, a′, ω′) to update the parameter ω, where r is the reward of the UAV;

[0054] Finally, a soft update method is adopted, i.e., θ′←τθ+(1+τ)θ′, ω′←τω+(1+τ)ω′ to update the current network parameters θ and ω to the Criticr target network and the Actor target network, where τ<<1 represents the update factor;

[0055] Among them: Actor target network represents: responsible for regularly updating the parameter θ to θ′, and then obtaining the next action a′ based on the state s′ in the experience pool;

[0056] The critic target network is responsible for regularly updating the parameter ω to ω′, and calculating Q′(s′, a′, ω′) to update the parameter ω of the current network.

[0057] Similarly, for different ship channel information, training is carried out according to the above training process to effectively solve the dynamic trajectory planning problem under different UAV take-off points and different ship channels.

[0058] In step 5, the initial position (x0, y0) of the UAV is randomly selected within the range of (0, [0, 300]) m, and the final position (x F ,y F )=(300,150)m, where (x0,y0) and (x F ,y F ) represent the horizontal coordinates of the initial position and the final position of the UAV respectively; the horizontal coordinate of the initial position of the mobile ship D (x d [0],y d [0]) = (0, 150) m; Assume that there are two eavesdroppers on the ocean, and their horizontal coordinates are and

[0059] In addition, it is assumed that the channel power gain p0 and the Rice factor K[n] are 40dBm and 31.3 respectively; the noise variance UAV transmission power Maximum flight speed of the UAV Maximum speed of the ship The flight altitude of the UAV is H = 50m. The parameters of the Q-learning algorithm are set as follows: learning rate α = 0.3, discount factor γ = 0.99, initial exploration rate ε = 1, and exploration rate decay factor κ = 0.99995. The parameters of the DDPG algorithm are set as follows: the size of the experience pool R b= 100000, the number of mini-batch samples N b =64, the learning rates of Actor and Critic networks are 10 -4 and 10 -3 .

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] 1. The reinforcement learning algorithms based on Q-learning and DDPG proposed in the present invention both take the initial position of the UAV into consideration. The initial position of the UAV is randomly selected before each trajectory optimization. Therefore, the two proposed algorithms can solve the problem of real-time planning of dynamic trajectories for different take-off points of the UAV.

[0062] 2. Compared with the existing traditional optimization algorithms, the reinforcement learning algorithms based on Q-learning and DDPG proposed in the present invention take the initial position of the UAV into account (the UAV take-off points are randomly distributed on the shore), and the initial position of the UAV is randomly selected before each trajectory optimization; at the same time, the reinforcement learning algorithms based on Q-learning and DDPG proposed in the present invention can learn the different channel information of the shipborne AIS, so that the UAV flight trajectory can be planned in real time according to the communication needs of different ships, effectively ensuring the real-time nature of marine communications. Therefore, the two algorithms proposed can solve the dynamic trajectory planning problems of different UAV take-off points and different ship channels.

[0063] 3. The reinforcement learning scheme proposed in the present invention is mainly aimed at ocean scenarios, in which marine users are not fixed or randomly distributed like land users. Marine ships have their own fixed routes and predictable movement patterns. At the same time, compared with land scenarios, the characteristics of sea channels are more easily affected by environmental factors such as sea surface conditions and meteorological conditions. The uniqueness of ocean propagation brings new channel characteristics, including scatterer sparsity, wave motion, ocean surface waveguide effect, and climate. Therefore, the reinforcement learning algorithm proposed in the present invention fully considers the constraints of the inherent ocean environment and solves the problem that existing algorithms cannot be directly extended to ocean scenarios. Compared with the UAV-ground free space loss model of land scenarios, the present invention proposes a UAV-ship channel modeling that combines large-scale and small-scale fading, which can better reflect the effects of wave motion and ocean scattering.

[0064] 4. The present invention realizes real-time planning of UAV trajectories in dynamic marine environments based on reinforcement learning algorithms, which greatly improves the performance and efficiency of marine communications. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0066] Figure 1 It is the overall flow chart of the present invention.

[0067] Figure 2 This is a flow chart of the UAV trajectory optimization based on the Q-learning algorithm of the present invention.

[0068] Figure 3 This is a flow chart of the UAV trajectory optimization based on the DDPG algorithm of the present invention.

[0069] Figure 4 These are the real-time planning results of UAV trajectories based on the Q-learning algorithm and the DDPG algorithm at different take-off positions of the present invention.

[0070] Figure 5 These are the real-time planning results of UAV trajectories based on the Q-learning algorithm and the DDPG algorithm under different ship tracks in the present invention.

[0071] Figure 6 This is the average reward curve obtained by each iteration based on the Q-learning algorithm and the DDPG algorithm of the present invention.

[0072] Figure 7 This is a curve diagram of average confidentiality rate obtained by different algorithms under different flight times of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0074] Please refer to Figure 1 , a real-time trajectory planning method for a sea area safety communication UAV based on reinforcement learning, the method comprising:

[0075] The method comprises the following steps:

[0076] Step 1: Establish a UAV-ship channel model combining large-scale fading and small-scale fading;

[0077] Step 2: Obtain the ship's channel information through the ship-borne AIS, and obtain the signal-to-noise ratio and rate received by the mobile ship and the eavesdropping node respectively;

[0078] Step 3: Obtain the optimal trajectory of the UAV by solving the maximum average confidentiality rate of the system;

[0079] Step 4: The UAV motion model based on discrete space and the UAV motion model based on continuous space use Q-learning and DDPG algorithms to optimize trajectories respectively;

[0080] Step 5: Set relevant simulation parameters;

[0081] Step 6. Display the real-time planning results of UAV trajectories of the two algorithms under different take-off positions, the real-time planning results of UAV trajectories of the two algorithms under different ship tracks, the average rewards obtained by each iteration of the two algorithms, and the average confidentiality rates obtained by different algorithms under different flight times.

[0082] In step 1, it is assumed that the flight time T of the drone S is divided into very small equal intervals d t Divided into N time slots, that is, T = Nd t ; In each time slot, considering that the actual sea channel characteristics are easily affected by sea surface conditions and meteorological conditions, the channel fading model of UAV-mobile ship or eavesdropping is modeled into a representation including large-scale fading and small-scale fading:

[0083]

[0084] Among them, g sd [n] is the channel power gain from UAV S to mobile ship D, From drone S to eavesdropping node E k The channel power gain of

[0085] In the formula, represents the distance from the UAV to the ship D, Indicates UAV to eavesdrop E k distance.

[0086] In step 2, the ship on the ocean travels along a pre-planned route, and the route information is obtained through the ship-borne AIS, so it is assumed that the route of ship D is:

[0087]

[0088] Among them, V d is the maximum shipping speed of the ship, d t represents the time slot, x d [n] represents the coordinate of ship D on the x-axis at time slot n, d [n+1] represents the x-axis coordinate of ship D at time slot n+1, y d [n] represents the coordinate of ship D on the y-axis at time slot n, d [n+1] represents the coordinate of ship D on the y-axis at time slot n+1, (xd [n],y d [n]) and (x d [n+1],y d [n+1]) represent the positions of the ship at time slots n and n+1 respectively;

[0089] The signal drying ratio and rate received by the mobile ship D are:

[0090]

[0091] Where: SINR represents signal-to-noise ratio, SINR d represents the signal-to-dryness ratio of ship D, R d represents the speed of ship D, P s [n] represents the transmit power of the UAV; represents the noise variance at the ship D;

[0092] The signal noise ratio and rate received by the eavesdropping node are:

[0093]

[0094] In the formula, SINR represents the signal-to-noise ratio, represents the signal-to-noise ratio of the eavesdropping node E, represents the noise variance at the eavesdropping E;

[0095] After optimizing the system, the maximum average confidentiality rate of the system is expressed as:

[0096]

[0097]

[0098] In the formula, It is the maximum horizontal flight speed of the UAV.

[0099] Please refer to Figure 2 In step 4, the sea area UAV trajectory optimization based on Q-learning:

[0100] First, a Q-learning framework is proposed to solve the discrete trajectory optimization problem. The Q-learning algorithm is regarded as a Markov decision process, which is represented as a set<S,A,R,P,γ> ;

[0101] S = {s1, s2, ....} represents the location or state set of the UAV in each time slot;

[0102] Among them, the state of the nth time slot can be expressed as: S n ={x s (n),y s(n),H},(x s (n),y s (n)) represents the horizontal coordinate of the UAV, and H represents the vertical coordinate of the UAV;

[0103] A represents the action set available to the UAV. In the discrete space under the Q-learning framework, there are 9 available actions {up, down, left, right, upper left, upper right, lower left, lower right, hover};

[0104] R is the reward function of the UAV, denoted as r n =r s +r b +r f ;

[0105] where r s represents the confidentiality rate reward, r b represents the penalty for UAV flying out of the boundary, r f represents the flight time penalty;

[0106] P represents the state transition probability, and P(s′|s, a) represents the probability distribution of the UAV moving to the next state s′ after taking action a in the current state s;

[0107] γ represents the discount factor, which determines whether the UAV values ​​future rewards or current rewards;

[0108] At the beginning of each iteration, a UAV take-off point on the shore is randomly selected for training, and since mobile ships follow fixed routes, the route information is obtained through the shipborne AIS historical information;

[0109] First, the UAV will select an action a according to the exploration rate ε in the nth time slot. n If the action causes the UAV to fly out of the boundary or fail to return on time, the UAV will be affected by r b and r f punishment,

[0110] Where: r b and r f is a negative constant;

[0111] Otherwise the UAV will follow a n Move to the next state s′ and get reward r n ;

[0112] Then according to the update formula:

[0113] Q π (s n , a n )←Q π (s n , a n)+α(r n +γmaxQ π (s n+1 , a n+1 )-Q π (s n , a n )) to update the Q table;

[0114] Among them, Q π (s,a)=Ε(R n =|s n =s,a n =a), which represents the expected future reward when the UAV follows the strategy π(a|s) and takes action a in state s;

[0115] represents the total discounted reward of the UAV in the nth time slot, represents the learning rate;

[0116] According to different ship channel information, training is carried out according to the above training process to effectively solve the dynamic trajectory planning problem under different UAV take-off points and different ship channels.

[0117] Please refer to Figure 3 In step 4, the sea area UAV trajectory optimization based on DDPG:

[0118] A DDPG framework is established to solve the continuous trajectory optimization problem. At the beginning of each iteration, a UAV take-off point on the shore is randomly selected for training. Since mobile ships sail along fixed routes, the route information is obtained through the ship-borne AIS historical information;

[0119] In each time slot, first add random noise n to the Actor current network N To avoid UAV falling into local optimization, action a = π θ (s)+n N ,here

[0120] Among them: Actor current network representation: responsible for updating the parameter θ through the gradient descent method, and taking the action a according to the current state s to interact with the environment to produce the next state s′ and reward r n ;

[0121] Secondly, also considering the issue of boundaries and flight time, when the action causes the UAV to fly out of the boundary or fail to return on time, the UAV will be affected by r b and r f The penalty is, where r b and r f is a negative constant;

[0122] Otherwise the UAV will follow a n Move to the next state s′ and get reward r n ; Then the corresponding element (s n , a n , r n ,s n+1 ) into the experience pool R b middle;

[0123] When the experience pool is full, batch sampling is performed from the experience pool and the loss function is used to calculate the and the gradient function To update the parameters θ and ω of the Critic current network and the Actor current network respectively, where N b is the number of samples in the mini-batch, and y n =r+γQ′(s′,a′,ω′);

[0124] Among them: Critic current network representation: responsible for calculating the current Q(s, a, ω) value, and according to the target value y n =r+γQ′(s′, a′, ω′) to update the parameter ω, where r is the reward of the UAV;

[0125] Finally, a soft update method is adopted, i.e., θ′←τθ+(1+τ)θ′, ω′←τω+(1+τ)ω′ to update the current network parameters θ and ω to the Criticr target network and the Actor target network, where τ<<1 represents the update factor;

[0126] Among them: Actor target network represents: responsible for regularly updating the parameter θ to θ′, and then obtaining the next action a′ based on the state s′ in the experience pool;

[0127] The critic target network is responsible for regularly updating the parameter ω to ω′, and calculating Q′(s′, a′, ω′) to update the parameter ω of the current network.

[0128] Similarly, for different ship channel information, training is carried out according to the above training process to effectively solve the dynamic trajectory planning problem under different UAV take-off points and different ship channels.

[0129] In step 5, the initial position (x0, y0) of the UAV is randomly selected within the range of (0, [0, 300]) m, and the final position (x F ,y F )=(300,150)m, where (x0,y0) and (x F ,y F ) represent the horizontal coordinates of the initial position and the final position of the UAV respectively; the horizontal coordinate of the initial position of the mobile ship D (x d[0],y d [0]) = (0, 150) m; Assume that there are two eavesdroppers on the ocean, and their horizontal coordinates are and

[0130] In addition, it is assumed that the channel power gain p0 and the Rice factor K[n] are 40dBm and 31.3 respectively; the noise variance The transmission power P of the UAV s [n] = 15dBm, maximum flight speed of the UAV Maximum speed of the ship The flight altitude of the UAV is H = 50m. The parameters of the Q-learning algorithm are set as follows: learning rate α = 0.3, discount factor γ = 0.99, initial exploration rate ε = 1, and exploration rate decay factor κ = 0.99995. The parameters of the DDPG algorithm are set as follows: the size of the experience pool R b = 100000, the number of mini-batch samples N b =64, the learning rates of Actor and Critic networks are 10 -4 and 10 -3 .

[0131] Please refer to Figure 4-7 , Figure 4 The initial position of the mobile ship D is represented by ☆, and × represents the eavesdropper E n , △ and ▽ represent the starting position and the end position of the UAV respectively. Since the take-off points of the UAV are randomly distributed on the shore in the actual scenario, the traditional algorithm cannot handle this dynamic take-off point situation. However, the Q-learning and DDPG algorithms we proposed both take the initial position of the UAV into account. The initial position of the UAV is randomly selected before each trajectory optimization. Therefore, both algorithms can solve the dynamic trajectory optimization problem of the UAV. Figure 4 From the simulation results shown, for any given UAV initial position (x0, y0) = (0, 210) m and (x0, y0) = (0, 90) m (for example, in reality, the UAV selected to provide communication services is not idle, and another UAV needs to be selected to provide services), both the Q-learning and DDPG algorithms can plan the trajectory of the UAV in real time. From the trajectory diagram, it can be seen that the UAV first chases the ship while avoiding the eavesdropper E1, then follows the ship for as long as possible, and finally, due to the flight time limit, the UAV will fly towards the end while avoiding the eavesdropper E2. This flight mode of the UAV indicates that it must avoid the eavesdropper while approaching the target, so as to maximize the safety rate as much as possible.

[0132] Figure 5The Q-learning and DDPG algorithms are shown in d [n],y d [n]) and speed The real-time planning diagram of the UAV trajectory is obtained under the simulation results. The ship's channel information can be obtained through the ship-borne AIS. From the simulation results, we can see that for the mobile ship 1 (solid line △, maximum sailing speed ) and mobile ship 2 (dashed line △, maximum sailing speed ), the UAV trajectories obtained by the Q-learning and DDPG algorithms are to follow the ship for a long time and then return while avoiding eavesdroppers as much as possible. By comparing the trajectories of the UAV in mobile ship 1 and mobile ship 2, we can find that when the UAV in mobile ship 2 passes two eavesdroppers, it does not follow the mobile ship as closely as before, but flies in the middle of the two eavesdroppers. This is mainly because flying in the middle of the eavesdroppers can keep away from both eavesdroppers as much as possible.

[0133] Figure 6 The figure shows the rewards obtained by the Q-learning and DDPG algorithms after each iteration. As we can see from the figure, as the number of iterations increases, the average confidentiality rate obtained by the two algorithms will first increase, and then gradually stabilize. In addition, by comparing the two algorithms, we can see that the DDPG algorithm converges significantly earlier than Q-learning. This is mainly because with continuous training, the Actor network gradually stabilizes, and the optimal action is obtained each time, which improves the training efficiency. In addition, we can also find that compared with the Q-learning algorithm, the confidentiality rate of the DDPG algorithm at the end of convergence is higher. This is mainly due to the fact that the DDPG algorithm allows the UAV to choose flight actions more freely, allowing the UAV to better approach mobile ships and provide better communication services.

[0134] Figure 7 The performance of the proposed Q-learning and DDPG algorithms is compared with the A3C algorithm and the traditional optimization algorithm (COA algorithm). Comparing the A3C and DDPG algorithms, we found that their performance is relatively close, and the DDPG algorithm is slightly better, because they both use the Actor-Critic framework. Compared with the Q-learning algorithm, the proposed DDPG algorithm is better, mainly because the DDPG algorithm can solve the trajectory optimization problem of the agent in continuous space, so that the UAV can better approach the mobile ship. Finally, we can see that the COA algorithm has poor performance compared to other reinforcement learning algorithms, mainly because the traditional optimization algorithm cannot adapt well to the highly dynamic marine environment.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time trajectory planning method for maritime safety communication drones based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Establish a UAV-ship channel model combining large-scale fading and small-scale fading; Step 2: Obtain the ship's channel information through the ship-borne AIS, and obtain the signal-to-noise ratio and rate received by the mobile ship and the eavesdropping node respectively; Step 3: Obtain the optimal trajectory of the UAV by solving the maximum average confidentiality rate of the system; Step 4: The UAV motion model based on discrete space and the UAV motion model based on continuous space use Q-learning and DDPG algorithms to optimize trajectories respectively; Step 5: Set relevant simulation parameters; Step 6: Display the real-time planning results of the UAV trajectory of the two algorithms under different take-off positions, the real-time planning results of the UAV trajectory of the two algorithms under different ship tracks, the average rewards obtained by each iteration of the two algorithms, and the average confidentiality rate obtained by different algorithms under different flight times; In step 4, the sea area drone trajectory optimization based on Q-learning: First, a Q-learning framework is proposed to solve the discrete trajectory optimization problem. The Q-learning algorithm is regarded as a Markov decision process, which is represented as a set<S,A,R,P,γ> ; S = {s1, s2, ....} represents the location or state set of the UAV in each time slot; Among them, the state of the nth time slot can be expressed as: S n ={x s (n),y s (n),H},(x s (n),y s (n)) represents the horizontal coordinate of the UAV, and H represents the vertical coordinate of the UAV; A represents the action set available to the UAV. In the discrete space under the Q-learning framework, there are 9 available actions {up, down, left, right, upper left, upper right, lower left, lower right, hover}; R is the reward function of the UAV, denoted as r n × s +r b +r f ; where r s represents the confidentiality rate reward, r b represents the penalty for UAV flying out of the boundary, r f represents the flight time penalty; P represents the state transition probability, and P(s′|s, a) represents the probability distribution of the UAV moving to the next state s′ after taking action a in the current state s; γ represents the discount factor, which determines whether the UAV values ​​future rewards or current rewards; At the beginning of each iteration, a UAV take-off point on the shore is randomly selected for training, and since mobile ships follow fixed routes, the route information is obtained through the shipborne AIS historical information; First, the UAV will select an action a according to the exploration rate ε in the nth time slot. n If the action causes the UAV to fly out of the boundary or fail to return on time, the UAV will be affected by r b and r f punishment, Where: r b and r f is a negative constant; Otherwise the UAV will follow a n Move to the next state s′ and get reward r n ; Then according to the update formula: Q π (s n , a n )←Q π (s n , a n )+α(r n +γmaxQ π (s n+1 , a n+1 )-Q π (s n , a n )) to update the Q table; Among them, Q π (s, a) = E(R n |s n =s,a n =a), which represents the expected future reward when the UAV follows the strategy π(a|s) and takes action a in state s; represents the total discounted reward of the UAV in the nth time slot, Represents the learning rate.

2. The real-time trajectory planning method for sea area safety communication drone based on reinforcement learning according to claim 1 is characterized in that: In step 2, the ship on the ocean travels along a pre-planned route, and the route information is obtained through the ship-borne AIS, so it is assumed that the route of ship D is: Among them, V d is the maximum shipping speed of the ship, d t represents the time slot, x d [n] represents the coordinate of ship D on the x-axis at time slot n, d [n+1] represents the x-axis coordinate of ship D at time slot n+1, y d [n] represents the coordinate of ship D on the y-axis at time slot n, d [n+1] represents the coordinate of ship D on the y-axis at time slot n+1, (x d [n],y d [n]) and (x d [n+1],y d [n+1]) represent the positions of the ship at time slots n and n+1 respectively; The signal drying ratio and rate received by the mobile ship D are: Where: SINR represents signal-to-noise ratio, SINR d represents the signal-to-dryness ratio of ship D, R d represents the speed of ship D, P s [n] represents the transmit power of the UAV; represents the noise variance at the ship D; The signal noise ratio and rate received by the eavesdropping node are: In the formula, SINR represents the signal-to-noise ratio, represents the signal-to-noise ratio of the eavesdropping node E, represents the noise variance at the eavesdropping point E; After optimizing the system, the maximum average confidentiality rate of the system is expressed as: In the formula, It is the maximum horizontal flight speed of the UAV.

3. The real-time trajectory planning method for sea area safety communication drone based on reinforcement learning according to claim 2 is characterized in that: In step 4, the sea area drone trajectory optimization based on DDPG: A DDPG framework is established to solve the continuous trajectory optimization problem. At the beginning of each iteration, a UAV take-off point on the shore is randomly selected for training. Since mobile ships sail along fixed routes, the route information is obtained through the ship-borne AIS historical information; In each time slot, first add random noise n to the Actor current network N To avoid UAV falling into local optimization, action a = π θ (s)+n N ,here Among them: Actor current network representation: responsible for updating the parameter θ through the gradient descent method, and taking the action a according to the current state s to interact with the environment to produce the next state s′ and reward r n ; Secondly, also considering the issue of boundaries and flight time, when the action causes the UAV to fly out of the boundary or fail to return on time, the UAV will be affected by r b and r f The penalty is, where r b and r f is a negative constant; Otherwise the UAV will follow a n Move to the next state s′ and get reward r n ; Then the corresponding element (s n ,a n , r n ,s n+1 ) into the experience pool R b middle; When the experience pool is full, batch sampling is performed from the experience pool and the loss function is used to calculate the and the gradient function To update the parameters θ and ω of the Critic current network and the Actor current network respectively, where N b is the number of samples in the mini-batch, and y n =r+γQ′(s′,a′,ω′); Among them: Critic current network representation: responsible for calculating the current Q(s, a, ω) value, and according to the target value y n =r+γQ′(s′, a′, ω′) to update the parameter ω, where r is the reward of the UAV; Adopt the soft update method, that is, θ′←τθ+(1+τ)θ, ω′←τω+(1+τ)ω′ to update the current network parameters θ and ω to the Criticr target network and the Actor target network, where τ<<1 represents the update factor; Among them: Actor target network represents: responsible for regularly updating the parameter θ to θ′, and then obtaining the next action a′ based on the state s′ in the experience pool; The critic target network is responsible for regularly updating the parameter ω to ω′, and calculating Q′(s′, a′, ω′) to update the parameter ω of the current network.

4. The real-time trajectory planning method for sea area safety communication drone based on reinforcement learning according to claim 1 is characterized in that: In step 5, the initial position (x0, y0) of the UAV is randomly selected within the range of (0, [0, 300]) m, and the final position (x F ,y F )=(300,150)m, where (x0,y0) and (x F ,y F ) represent the horizontal coordinates of the initial position and the final position of the UAV respectively; the horizontal coordinate of the initial position of the mobile ship D (x d [0],y d [0]) = (0, 150) m; Assume that there are two eavesdroppers on the ocean, and their horizontal coordinates are and In addition, it is assumed that the channel power gain p0 and the Rice factor K[n] are 40dBm and 31.3 respectively; the noise variance The transmission power P of the UAV s [n] = 15dBm, maximum flight speed of the UAV Maximum speed of the ship The flight altitude of the UAV is H = 50m. The parameters of the Q-learning algorithm are set as follows: learning rate α = 0.3, discount factor γ = 0.99, initial exploration rate ε = 1, and exploration rate decay factor κ = 0.99995. The parameters of the DDPG algorithm are set as follows: the size of the experience pool R b = 100000, the number of mini-batch samples N b =64, the learning rates of Actor and Critic networks are 10 -4 and 10 -3 .

Images