A method for physical layer security transmission of a UAV-assisted ISAC system based on reinforcement learning

Abstract

The application is a kind of UAV-assisted ISAC system physical layer security transmission method based on reinforcement learning, aiming at dealing with the information leakage risk under the condition of multi-sensing target and imperfect channel state information. It includes: constructing a UAV-assisted ISAC system model, and further constructing a secure communication model and a sensing model based on the model; a target function aiming at maximizing the user safety rate is established; in order to solve the optimization problem proposed, a double-delay deep deterministic policy gradient algorithm of generative diffusion model is proposed to solve. Compared with the traditional algorithm, the application enhances the exploration ability by using GDM, effectively balances the relationship between exploration and utilization, improves the system performance and enhances the robustness. The application takes the user and safety rate as the index, and verifies the effectiveness of the method in the simulation results, and shows the superior secrecy performance and adaptability to the environment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of communication and information security, and specifically relates to a physical layer security transmission method for multiple sensing targets and imperfect CSI in an unmanned aerial vehicle (UAV) assisted integrated sensing and communication system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and can constitute the prior art. During the implementation of the present application, the inventors found at least the following problems in the prior art.

[0003] With the rapid growth of demand for intelligent unmanned aerial vehicles (UAVs), their application scenarios in terms of communication support and sensing detection functions are becoming increasingly widespread. However, the current traditional UAV system designed based on the separation of communication and sensing has been unable to meet the increasingly complex requirements, and more efficient solutions are urgently needed. Integrated sensing and communication (ISAC) technology, as an emerging approach, can improve the spectrum utilization efficiency and hardware resource utilization of the UAV system by implementing communication and sensing functions in the same frequency band, thereby effectively addressing the limitations of traditional design.

[0004] Under this background, UAVs, with their high mobility and rapid deployment capability, can flexibly act as temporary base stations or relay nodes, working in coordination with ISAC technology, not only providing efficient communication support for ground users, but also achieving extensive sensing. However, due to the openness and broadcast nature of wireless channels, especially the significant characteristics of the line-of-sight (LoS) channel between UAVs and ground nodes, UAV communication links are more vulnerable to threats from ground eavesdroppers, severely reducing the security of information transmission. In addition, the pointing of the transmit beam towards the target during the sensing process further exacerbates the risk of information leakage. Existing researches mostly focus on single sensing target scenarios, assuming that communication users and sensing targets are stationary and have perfect channel state information (CSI). These idealized assumptions are difficult to correspond to the actual environment. In addition, traditional convex optimization methods cannot effectively solve the problem of secure transmission under dynamic environmental changes, while deep reinforcement learning (DRL) has dynamic environmental adaptability, but conventional algorithms have large variance and policy bias problems in high-dimensional action spaces.

[0005] As application No. 202410215216.3 patent name is "a physical layer security transmission method and system in a sense integration unmanned aerial vehicle network", its method is: constructing a sense integration unmanned aerial vehicle scene; modeling the sense signal transmission and sensing process of the unmanned aerial vehicle; modeling the secure communication process of the unmanned aerial vehicle and the ground user; modeling the communication performance constraint, sensing beam pattern constraint and transmission power constraint of the unmanned aerial vehicle; establishing a secure communication beam forming optimization target, and solving by using the semi-positive optimization algorithm of the Dingkelbach algorithm. However, this patent does not consider the case of multiple sensing targets and non-perfect channel state information (CSI), which is difficult to cope with the actual environment, resulting in poor system security performance in complex scenarios and increasing the risk of information leakage. Moreover, due to the influence of non-perfect CSI on sense performance, the robustness of the system is also reduced.

[0006] For the problem of secure transmission in an unmanned aerial vehicle (UAV) assisted sense integration (ISAC) system with multiple sensing targets and non-perfect channel state information (CSI), how to develop a robust and intelligent physical layer security transmission method is a problem that needs to be solved in the field. SUMMARY

[0007] To solve some of the problems in the prior art, or at least alleviate these problems.

[0008] A UAV-assisted ISAC system physical layer security transmission method based on reinforcement learning, comprising the following steps:

[0009] Constructing a UAV-assisted ISAC system transmission model, and further constructing a secure communication model and a sensing model based on the UAV-assisted ISAC system transmission model;

[0010] Based on the UAV-assisted ISAC system transmission model, the secure communication model and the sensing model, under the constraints of total transmission power, communication user signal-to-interference noise ratio and sensing target beam pattern gain performance, a system optimization model is established by jointly designing the transmission beamforming, artificial noise and UAV trajectory to maximize the user and security rate; the optimization problem is expressed as:

[0011]

[0012] Wherein, the decision variable is the beamforming vector of the transmitted signal R v [n] is the artificial noise covariance matrix, q u [n]=(x u [n],y u [n] is the horizontal position of the UAV itself (i.e. the trajectory of the UAV); is the worst case security rate, Pmax is the maximum transmit power of the UAV; P(θ m [n])=a H (θ m [n])R x [n]a(θ m [n]) is a perception model, θ m [n] is the departure angle with error, R x [n] is the covariance matrix of the transmitted signal; a(θ m [n]) is the array steering vector, representing the steering vector of the mthperception target; Γ m is the minimum beam pattern gain, Γ c is the minimum communication signal-to-interference noise ratio; is the received signal-to-interference noise ratio at the legitimate user k; h k [n] is the channel vector from the UAV to the legitimate user k in the nth time slot, Δh k [n] represents the estimation error of the channel; V max is the maximum flight speed of the UAV in a time slot, is the initial position set for the UAV; x min , x max , y min , y max represent the boundaries of the region, (x u [n], y u [n], H) represents the time-varying position of the UAV in the nth time slot;

[0013] A generative diffusion model (GDM) is integrated into an actor network of a TD3 algorithm (double-delay deep deterministic policy gradient), and a GDMTD3 algorithm (double-delay deep deterministic policy gradient supporting generative diffusion model) is designed to solve the optimization problem to generate an optimal action according to a current environment state.

[0014] A transmission model of a UAV-assisted ISAC system is constructed, and a safety communication model and a perception model are further constructed based on the model, including the following steps:

[0015] A transmission model of a UAV-assisted ISAC system is constructed: one mobile unmanned aerial vehicle (UAV) serves as a dual-function base station and communicates with a legitimate user k; a perception target m (eavesdropper) eavesdrops on the information of the legitimate user; the UAV position changes in multiple time slots n;

[0016] A transmission signal model is established: in the nth time slot, the downlink transmission signal of the UAV is wherein represents a signal sent to the legitimate user k, the vector is a precoding vector of the transmitted signal, representative embedded artificial noise (AN), is the AN covariance matrix, H represents the fixed flight height of the UAV;

[0017] Establish a communication user imperfect CSI model: model the CSI of the legitimate user and the sensing target, including the legitimate user channel and the eavesdropping channel, and model the channel error and angle error.

[0018] Further, the UAV is equipped with an L-element uniform linear array for transmitting beamforming; the legitimate user and the sensing target are both single antennas.

[0019] Build a secure communication model, including defining the security rate of the legitimate user; the expression of the security rate in the worst case is given as:

[0020]

[0021] where [x] + =max{x,0}; is the eavesdropping rate of the eavesdropper, is the eavesdropping SINR of the mth target to the kth user; the achievable transmission rate of the legitimate user;

[0022] Build a sensing model of the system, including sensing the target through the transmitted waveform, using the beam pattern gain to represent the sensing performance, which is expressed as:

[0023] P(θ m [n])=a H (θ m [n])R x [n]a(θ m [n])

[0024] where a(θ m [n]) represents the steering vector of the mth sensing target.

[0025] Integrate the generative diffusion model (GDM) into the Actor network of the TD3 algorithm (double-delay deep deterministic policy gradient), and design a GDMTD3 algorithm (double-delay deep deterministic policy gradient supporting generative diffusion model) to solve the optimization problem, including the following steps:

[0026] Model the optimization problem as a Markov decision process, and define the state space action space reward function ;

[0027] By integrating GDM into the Actor network of the TD3 algorithm, complex state features are captured, and the optimal action is generated based on the current environmental state.

[0028] Furthermore, the state space definition The state of time slot n consists of the location of the UAV, the location information of legitimate ground users and sensing targets (eavesdroppers), and channel state information; s[n] = {q u [n],u k [n],u m [n],h k [n],g m [n]}, Where q u [n] is the position of the UAV, u k [n]、u m [n] represents the location information of the legitimate ground user and the sensing target (eavesdropper), respectively; h represents the location information of the eavesdropper. k [n]、g m [n] are the channel vectors from the UAV to the legitimate user and the perceived target (eavesdropper), respectively;

[0029] The action space Defined as the actions within time slot n, including the transmitted beamforming vector, the artificial noise covariance matrix, and the UAV flight angle and distance; a[n] = {w1[n],...w K [n],R v [n],φ[n],dis[n]}, where w k [n] is the transmitted signal beamforming vector, R v [n] is the covariance matrix of the artificial noise, and φ[n] and dis[n] represent the direction and distance of the UAV displacement, respectively;

[0030] The reward function In the nth time slot, the instant reward can be defined as: Where r p [n] represents the penalty imposed when a UAV flies out of the boundary, and ω is the penalty factor.

[0031] Furthermore, the optimization problem is modeled as a Markov decision process, including the following steps:

[0032] A UAV is an agent; during each training time period, the agent first obtains the state s[n] from the environment;

[0033] Then select strategy a[n] from the action space;

[0034] Again, the environment updates the current state to s[n+1] and obtains the corresponding reward r[n];

[0035] The experience tuple s[n], a[n], r[n], s[n+1] is then stored in the replay memory buffer .

[0036] The GDMTD3 algorithm is designed, which also includes using a parametric model to approximate the conditional distribution The parametric model is represented as:

[0037]

[0038] where, is the mean, and g is the conditional information; is a predetermined variance factor, represented as represents the alpha of all steps r≤t r Cumulative product, where alpha t = 1-beta t ;

[0039] The average value of the reverse process is calculated as:

[0040]

[0041] where, is the original data; t is the diffusion step number, is the distribution of the t-th step, I is the standard Gaussian noise, and beta t is the variance factor;

[0042] Including the following steps:

[0043] Estimating the original data is represented as:

[0044]

[0045] where, is a deep neural network;

[0046] According to the conditional information g, the de-noising noise is generated, and then the mean is indirectly approximated, represented as:

[0047]

[0048] The reverse transformation from to is tracked to establish the generation distribution As follows:

[0049]

[0050] where, denotes the standard normal distribution; denotes the conditional distribution of under the known de-noising to

[0051] When the distribution is successfully trained, continue to obtain samples from the above formula

[0052] The GDMTD3 algorithm is designed, and a reparameterization process that is beneficial to sampling is also included, which is represented as follows:

[0053]

[0054] where s represents the current state of the environment in DRL, as a conditional variable in the parameterized function; is the operator of Hadamard product, denotes the standard Gaussian noise.

[0055] A computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the UAV-assisted ISAC system physical layer security transmission method based on reinforcement learning.

[0056] The present application has the following beneficial effects:

[0057] 1. Joint optimization of beamforming and UAV trajectory: by jointly optimizing the UAV trajectory, transmit beamforming and artificial noise covariance matrix, the present application realizes the maximization of system and safety rate, and significantly improves the adaptability of the system in dynamic environment.

[0058] 2. Algorithm innovation and performance improvement: the present application combines GDM with TD3 algorithm, effectively enhances the exploration ability of the algorithm, optimizes the balance between exploration and utilization, and thus significantly improves the decision efficiency. At the same time, the proposed GDMTD3 algorithm exhibits stronger robustness and higher learning efficiency in high-dimensional action space.

[0059] 3. Adaptability: the designed method can quickly respond to the dynamic changes of user and perceived target position, and realizes the dynamic adjustment of UAV under the conditions of multiple targets and imperfect CSI. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a scene diagram of the UAV-assisted ISAC system of the present application;

[0061] Figure 2 is a flowchart of the present application;

[0062] Figure 3A UAV flight trajectory diagram in a dynamic scenario for the present application;

[0063] Figure 4 A comparison diagram of the safe rate of the scheme proposed in the present application at different powers and other schemes;

[0064] Figure 5 A comparison diagram of the safe rate of the scheme proposed in the present application at different perception thresholds and other schemes. DETAILED DESCRIPTION

[0065] The embodiments of the present application are only used to illustrate the present application and not to limit the present application, and various replacements and changes can be made according to the ordinary technical knowledge and conventional means in the art without departing from the technical thought of the present application, which should be included in the scope of the present application.

[0066] To solve the problems existing in the prior art, the present application proposes a physical layer secure transmission method for multiple perception targets and imperfect CSI in a UAV assisted ISAC system (i.e. a physical layer secure transmission method for a UAV assisted ISAC system based on reinforcement learning), which maximizes the user and safe rate by jointly designing the transmit beamforming, artificial noise and UAV trajectory under the constraints of total transmit power, communication user signal-to-interference-and-noise ratio and perception target beam pattern gain performance. To achieve the above purpose, the present application adopts the technical scheme as shown in Figure 2

[0067] S1: constructing a UAV assisted ISAC system transmission model, and further constructing a secure communication model and a perception model based on the UAV assisted ISAC system transmission model.

[0068] Comprising the following steps:

[0069] S11: constructing a UAV assisted ISAC system transmission model

[0070] The system model structure composition is as shown in Figure 1 A UAV-ISAC system is considered, in which a mobile unmanned aerial vehicle (UAV) serves as a dual-function base station, broadcasting K independent confidential information to K legitimate ground users, and M perception targets (potential eavesdroppers) eavesdropping all user information; the UAV position changes in multiple time slots n.

[0071] The UAV is equipped with an L-element uniform linear array for transmit beamforming, and the users and perception targets are both single-antenna. In order to reasonably analyze the system, the total service time is divided into N time slots, and it is assumed that the position of the UAV is approximately unchanged in each time slot. In the nth time slot, the position of the legitimate user k is defined as u k [n]=(x​k [n],y k [n], 0). The position of the perceived target m is defined as u. m [n] = (x m [n],y m [n], 0), let (x u [n],y u [n],H) represents the time-varying position of the UAV in time slot n, and the horizontal position of the UAV itself is q. u [n] = (x u [n],y u [n]), where H represents its fixed flight altitude. The deployment location constraints of a UAV can be expressed as: x min ≤x u [n]≤x max ,y min ≤y u [n]≤y max , where x min ,x max ,y min ,y max Indicates the boundary of a region. V max For the maximum flight speed of the UAV within a time slot, there is a constraint ||q u [n]-q u [n-1]||≤V max ,

[0072] S12: Establish the transmission signal model

[0073] exist Figure 1 Under the system model shown, the signal model of the system is established. In time slot n, the downlink transmit signal of the UAV is given by the following formula:

[0074]

[0075] In the formula This indicates a signal sent to user k. It is the beamforming vector transmitted in time slot n. This represents the embedded AN. The covariance matrix of the transmitted signal is... in, It is the covariance matrix of the transmitted beamforming vector. Let AN be the covariance matrix.

[0076] The average transmit power of the UAV in the nth time slot is The maximum transmission power of the UAV is P. max Therefore, the total transmit power constraint is:

[0077] S13: Establishing a communication user imperfect CSI model

[0078] A communication user imperfect CSI and statistical CSI model is constructed. The legitimate user and sensing target CSI are modeled, including the legitimate user channel and eavesdropping channel, and the channel error and angle error are modeled. Since the height of the UAV is relatively high, there is usually a strong LoS link between the UAV and each ground user. Therefore, the present application considers a LoS channel model, on the basis of which the channel vector from the UAV to the legitimate user k is denoted as:

[0079]

[0080] wherein, is the estimated channel, Δh k [n] represents the estimation error of the channel. The estimated channel is wherein β c is the path loss at a reference distance of 1 m of the legitimate channel, η is the path loss factor from the UAV to the LoS link, is the distance from the UAV to the legitimate user k; a(θ k [n]) is the array steering vector, which is denoted as:

[0081]

[0082] wherein λ and d are the carrier wavelength and the spacing between two adjacent antennas, respectively, denotes the angle of departure corresponding to the user k; j represents an imaginary unit, L is the number of antennas, and T represents transposition.

[0083] A common estimation error is considered, i.e., a bounded error, i.e., an additive channel error model, which is modeled as ||Δh k [n]||≤μ k wherein ||·|| represents a modulus operation, μ k is a scalar representing the upper bound of the error, and the channel of the communication user is a bounded uncertainty. Limited by hardware capabilities and other factors, the UAV can obtain a rough CSI, and the channel can be modeled as a bounded CSI error model, which is a relatively bad assumption.

[0084] Then, a sensing target angle uncertainty model is established. In actual operation, it is not possible to know the accurate position of the target in advance. Therefore, a target position angle uncertainty region is defined. The sensing channel is modeled as a pure LoS channel, wherein β e is the path loss at a reference distance of 1 m of the eavesdropping channel, is the distance from the UAV to the sensing target m. The angle error is: wherein Δθ m [n]≤εm , ε m denotes the upper bound of the bounded error.

[0085] S14: Establishing the secure communication model

[0086] At the nth time slot, the received signal of the legitimate user k is denoted as where z k [n] is additive Gaussian noise, and the received Signal-to-Interference-plus-Noise Ratio (SINR) at the legitimate user k is:

[0087]

[0088] where denotes the communication additive Gaussian noise power, and the achievable transmission rate of the legitimate user is w i [n] denotes the transmit beamforming vector for the ith user.

[0089] The received signal at the mth sensing target is Similarly, where z m [n] is additive Gaussian noise, and the eavesdropping SINR of the mth target eavesdropping the kth legitimate user is given by:

[0090]

[0091] where denotes the eavesdropping additive Gaussian noise power, and tr(·) denotes the trace of a matrix, and the eavesdropping rate of the eavesdropper is

[0092] The achievable secure rate at the legitimate user is defined as the difference between the achievable rates at the legitimate user and the eavesdropper. Therefore, the expression of the secrecy rate in the worst case is given as where [x] + = max{x, 0}.

[0093] S15: Constructing the sensing model of the system

[0094] The above transmitted signals are used for both sensing and communication functions simultaneously, where each transmitted signal is considered as a radar pulse. Then, the sensing performance is characterized using the beam pattern gain, denoted as:

[0095] P(θ m [n]) = a H (θ m [n]) R x [n] a(θ m [n]) (6)

[0096] where a (θ m [n]) represents the steering vector of the mth sensing target.

[0097] S2: Establish a system optimization model to maximize user and safety rates

[0098] Based on the UAV-assisted ISAC system transmission model, the safety communication model and the sensing model, the present application proposes to establish a system optimization model by jointly designing the transmit beamforming, artificial noise and UAV trajectory under the constraints of total transmit power, communication user signal-to-interference-and-noise ratio and sensing target beam pattern gain performance, so as to maximize the user and safety rates. The optimization problem is expressed as:

[0099]

[0100] where the decision variable is the beamforming vector of the transmit signal R v [n] is the artificial noise covariance matrix, q u [n] = (x u [n], y u [n]) is the horizontal position of the UAV (i.e. the trajectory of the UAV); is the worst-case safety rate, P max is the maximum transmit power of the UAV; P(θ m [n]) = a H (θ m [n]) R x [n] a(θ m [n]) is the sensing model, θ m [n] is the angle error, R x [n] is the covariance matrix of the transmit signal; a(θ m [n]) is the array steering vector, representing the steering vector of the mth sensing target; Γ m is the minimum beam pattern gain, Γ c is the minimum communication signal-to-interference-and-noise ratio; is the signal-to-interference-and-noise ratio received at the legitimate user k; h k [n] is the channel vector from the UAV to the legitimate user k in time slot n, Δh k [n] represents the estimation error of the channel; V max is the maximum flight speed of the UAV in a time slot, is the initial position set for the UAV; x min , x max , y min , y max represent the boundaries of the region, (x u [n], y u[n], Hn represents the time-varying position of the UAV at time slot n. Constraint C1 is the total transmit power constraint of the UAV at time n, constraint C2 is the constraint of the sensing performance, which guarantees that the minimum beam pattern gain of multiple sensing targets can be met m . Constraint C3 is the communication performance constraint, which guarantees that each user can meet the minimum communication signal-to-interference noise ratio c . Constraints C4-C5 are constraints on the flight position of the UAV, which guarantee that the UAV starts from the set initial position , and does not fly out of the set area.

[0101] S3: design GDMTD3 algorithm to solve the optimization problem

[0102] The generative diffusion model (GDM) is integrated into the Actor network of the TD3 algorithm (twin delayed deep deterministic policy gradient), and the GDMTD3 algorithm (generative diffusion model-enabled twin delayed deep deterministic policy gradient) is designed to solve the optimization problem, so as to generate the optimal action according to the current environment state.

[0103] In order to effectively solve the planned problem, the method of DRL is used to solve the formulated problem. In order to adapt to the continuous state and action space, the TD3 learning method is selected from many variants of DRL. However, the conventional TD3 algorithm usually uses stacked fully connected layers in the Actor network, which is difficult to capture deeper data features. Therefore, these algorithms can cause the learned policy distribution to deviate from the real data distribution. GDM can better capture the complex dynamics of the problem scene, thereby generating a better solution strategy. Therefore, in view of the problems of low sampling efficiency, difficulty in modeling complex environment and slow convergence speed existing in the conventional TD3 algorithm, the TD3 algorithm is combined with the generative diffusion model, and a generative diffusion model-enabled twin delayed deep deterministic policy gradient (GDMTD3) method is proposed, which aims to deal with the presence of mobile users and eavesdroppers and non-perfect CSI. Since GDM has strong ability to model complex data distribution, it can reduce the number of samples required and more effectively represent the features of complex and high-dimensional state and action space; and its unique structure involves a diffusion process, which can provide a more stable and effective learning process, which can effectively solve the above problems. Specifically, the designed GDMTD3 algorithm includes the following steps:

[0104] First, the original optimization problem is modeled as a Markov decision process.

[0105] The original UAV-ISAC security problem can be modeled as a Markov Decision Process. The UAV is an Agent, where the system's state at each time slot depends only on the current state of the system and the action taken by the UAV. The interaction of the Agent with the environment is as follows, at each time period of the training set, the Agent first obtains the state s[n] from the environment, and then selects a policy a[n] from the action space; then, the environment updates the current state to s[n+1] and obtains the corresponding reward r[n]. The experience tuple s[n], a[n], r[n], s[n+1] is then stored in the replay memory buffer The corresponding elements are defined as follows:

[0106] State space The set of all possible observations of the system state by the UAV, i.e., the state space. Define as the state at time slot n; s[n] = {q u [n], u k [n], u m [n], h k [n], g m [n]}, where q u [n] is the position of the UAV, u k [n], u m [n] are the position information of the ground legitimate users and eavesdroppers, respectively, h k [n], g m [n] are the channel vectors from the UAV to the users and eavesdroppers, respectively.

[0107] Action space The UAV selects the corresponding action according to the observed state information. Define as the action at time slot n, a[n] = {w1[n],... w K [n], R v [n], φ[n], dis[n]}, where w k [n] is the transmit signal beamforming vector, R v [n] is the covariance matrix of artificial noise, φ[n], dis[n] represent the direction and distance of the UAV displacement, respectively.

[0108] Reward function The reward function is used to measure the performance of the agent and guide its decision-making process. In the optimization problem, the optimization goal is to maximize the system user and security rates under the constraints of sensing performance, communication performance, and transmit power. Therefore, in time slot n, the immediate reward is defined as where r p[n] the penalty for the UAV flying out of the boundary, ω is the penalty factor, the degree of punishment is proportional to the gap between the agent and the constraint, and is defined as follows:

[0109]

[0110] The final reward is That is, the purpose of the TD3 algorithm is to maximize the long-term cumulative discounted reward, where γ∈(0,1) is the discount factor.

[0111] Secondly, the GDM is integrated into the Actor network of the TD3 algorithm to capture complex state features and generate optimal actions according to the current environment state.

[0112] In view of the problems of low sampling efficiency, difficulty in modeling complex environment, and slow stability and convergence speed in the conventional TD3 algorithm, the GDM is integrated into the Actor network of the TD3 algorithm. The TD3 algorithm is an Actor-Critic-based DRL algorithm, wherein the Actor network outputs a deterministic action, and the Critic network evaluates the action state value function. The Actor network is an "actor network" or "policy network"; it is responsible for "decision making" and generates actions according to the current state by maximizing the Q value of the Critic. The Critic network is an "evaluator network" or "value network"; it is responsible for evaluating the value of the action generated by the Actor, providing a feedback signal, and updating its own parameters by minimizing the temporal difference error.

[0113] The goal is to find the optimal policy that maximizes the expected cumulative reward. The training process of TD3 includes updating the Actor and Critic networks based on a specific loss function. The update of the Critic network is achieved by minimizing the temporal difference error loss function, and a batch of samples is randomly extracted from the experience replay buffer with a size of B, and the loss function of the Critic network is approximated as:

[0114]

[0115] where is the parameter related to the Critic network, and the Bellman equation provides a recursive decomposition to update the action value function. The update frequency of the Actor network is lower than that of the Critic network to ensure stable learning. The goal of the Actor network is to maximize the expected Q value evaluated by the first Critic network. A batch of samples with a size of B is randomly extracted from the experience replay buffer , and the loss function of the Actor network is approximated as:

[0116]

[0117] The target network is updated using a soft update mechanism, which combines the parameters of the main network and the target network using weight factors. The update is defined as follows:

[0118]

[0119] as well as

[0120]

[0121] Here, τ is a small soft weighting factor. It can be seen that the updated parameters of the target network are a weighted combination of its original parameters and the corresponding network parameters.

[0122] Secondly, the inverse denoising process of GDM is utilized. Specifically, the inverse denoising process reconstructs the original data by systematically removing noise; in the inverse process, the goal is to remove noise iteratively from data following a standard Gaussian distribution. of Recovering original data from noisy samples However, statistical distribution This involves calculating data distributions, which is often difficult to handle in practice. The strategy is to use a parametric model. To approximate the conditional distribution This model can be represented as In the formula, Let g be the mean, where g represents conditional information. The predetermined variance factor is expressed as: in, α represents all steps r≤t. r Cumulative product, where α t =1-β t Using Bayes' theorem, the reverse process is reconstructed as a Gaussian probability density function. The average value of the reverse process is calculated as follows: Nevertheless, parametric models Unable to access Therefore, it is necessary to estimate it. in It is a deep neural network that generates denoised noise based on condition g, and then indirectly approximates the mean.

[0123]

[0124] Tracking from arrive The reverse transformation can establish a generative distribution. as follows:

[0125]

[0126] where denotes the standard normal distribution. Once the distribution is successfully trained, it can continue to draw samples from the above equation

[0127] The generative capability of the diffusion model can create a complex set of actions, which are refined through the reverse process of learning, so that actions can be directly sampled from the generative distribution. A major challenge in integrating diffusion models is to manage the random component, which makes the gradient descent method commonly used for training complicated. To overcome this problem, a reparameterization process that facilitates sampling is adopted, which is represented as follows:

[0128]

[0129] where s denotes the current state of the environment in DRL, as a conditional variable in the parameterized function. ⊙ is the operator of Hadamard product, denotes the standard Gaussian noise. Based on the above design, an action sampling algorithm based on GDM is designed and integrated into the existing TD3 algorithm.

[0130] The main steps of the action sampling process based on the generative diffusion model are detailed in Algorithm 1.

[0131] Table 1 Action sampling algorithm based on generative diffusion model

[0132]

[0133] Integrating GDM into the Actor network of the TD3 algorithm captures complex state features and generates optimal actions based on the current environment state, enhancing the decision-making ability of the Actor network. Algorithm 2 describes the specific implementation of this process.

[0134] Table 2 TD3 algorithm based on generative diffusion model

[0135]

[0136]

[0137] The computational complexity of GDMTD3 in the training phase is where denotes the number of parameters in the two online Critic networks, |θ d | denotes the number of parameters in the online actor network supporting diffusion. M represents the number of training rounds, N is the number of steps per round, d tis the number of denoising steps required to sample an action in a diffusion actor network. V represents the complexity of interacting with the environment, and d is the frequency of policy updates.

[0138] To demonstrate the effectiveness of the proposed scheme, a comprehensive evaluation of the proposed method is conducted, and the effectiveness and robustness of the proposed GDMTD3 in solving the UAV-ISAC security problem are verified. This example provides a description of the simulation settings, including environmental details, model design, and benchmarks for evaluating the performance of the proposed method. In this invention, a sensing scenario consisting of 1 independent UAV (equipped with 8 antennas), 3 moving legitimate users, and 2 moving sensing targets (eavesdroppers) is considered, with a UAV flight height of 100 m. To fully demonstrate the advantages of the proposed scheme, the following comparative schemes are set up:

[0139] Comparative scheme 1: UAV trajectory is fixed, and the beamforming and AN joint optimization strategy of the proposed algorithm is used.

[0140] Comparative scheme 2: without embedding AN, using the UAV trajectory and beamforming joint optimization strategy of the proposed algorithm.

[0141] Comparative scheme 3: using the conventional TD3 algorithm for UAV trajectory, beamforming, and AN joint optimization strategy.

[0142] Through comparative simulation with the above schemes, this example gives specific simulation and conclusions.

[0143] Figure 3 The UAV flight trajectory under the ground user dynamic change method is shown. The initial position of the UAV is [0, 0] meters, and the flight height is 100 meters. The UAV adjusts the path according to real-time sensing to optimize communication quality and avoid eavesdropping risks. Due to the dynamic change of the positions of ground users and eavesdroppers over time, the UAV flexibly adjusts the trajectory to cope with the dynamically changing environment. The UAV deployment position meets the sensing demand and improves the security rate, demonstrating its ability to adaptively adjust.

[0144] Figure 4 The trend of user and security rate (SSR) with power change is shown. Under the condition of fixed sensing target position and 3 dB sensing threshold, the proposed scheme is compared with the benchmark scheme. The results show that with the increase of power, the SSR of all schemes increases, and the performance of the proposed scheme is significantly better than that of the benchmark scheme, demonstrating its advantages in optimizing channel conditions and avoiding eavesdropping. In addition, the proposed scheme requires lower power to achieve the same SSR, proving its efficiency. Under the condition of imperfect CSI, the proposed scheme can achieve an SSR close to perfect CSI, indicating that the algorithm has good robustness and can effectively reduce the impact of CSI errors.

[0145] Figure 5 The trend of SSR with the change of the sensing threshold is demonstrated. The SSR performance of the proposed scheme and the baseline scheme is compared under different sensing thresholds. The results show that the SSR of all schemes decreases as the sensing threshold increases, because higher sensing threshold requires more power to be allocated for sensing task, reducing the communication power resource. The SSR of the proposed scheme is always better than that of the baseline scheme, which reflects its advantages in optimizing UAV path, power allocation and AN interference strategy. In addition, when there is uncertainty in the sensing target position, the SSR of the proposed algorithm is close to the perfect sensing condition, which shows that it has good robustness and can adapt to the case of slight uncertainty in the sensing target position.

[0146] The present application specifically includes:

[0147] A transmission model of the UAV-assisted ISAC system is constructed: the model contains multiple sensing targets, and the mobility of ground users and sensing targets is considered. By jointly optimizing the transmit beamforming, artificial noise covariance matrix and UAV trajectory, the sum rate and safety rate of users are maximized. First, the ideal scenario of known perfect communication user CSI and known accurate target position is considered, and the ideal performance upper bound is obtained by designing a solving algorithm. Then, the case of error in the CSI of legitimate users and angle error in the sensing target is considered. In order to solve the formulated optimization problem, a Generative Diffusion Model-enabled Twin Delayed Deep Deterministic Policy Gradient (GDMTD3) method is proposed, which integrates GDM into the Actor network of the TD3 algorithm. Since GDM can model complex data distribution and generate high-quality samples, GDMTD3 can effectively solve the problems of low sampling efficiency, difficulty in modeling complex environment, and slow stability and convergence speed of the conventional TD3 algorithm. Finally, the effectiveness of the proposed scheme is verified through simulation.

[0148] The present application combines Generative Diffusion Model (GDM) and Twin Delayed Deep Deterministic policy gradient (TD3) technology to solve the optimization problem under multiple sensing targets and imperfect CSI, maximize the system user and safety rate, reduce the risk of information leakage, and improve the system safety performance in complex scenarios. It effectively handles the impact of imperfect CSI on sensing performance and improves the robustness of the system.

[0149] The application is suitable for the field of unmanned aerial vehicle (UAV) communication, especially complex scenarios such as UAV operation and emergency communication in emergency situations. The technical solution ensures the dual efficiency of communication security and information perception through joint optimization of the flight trajectory of the UAV and wireless resources. In the field of military and security, the technical solution can significantly improve the flexibility and security of UAVs in heavy tasks such as military operations and border monitoring, enabling UAVs to dynamically adjust flight paths and resource management according to real-time battlefield situations, achieving safe and efficient information transmission and situation awareness. What is particularly important is that the application designs a highly adaptive physical layer security transmission strategy for the ubiquitous non-perfect CSI and dynamic environmental changes in the actual communication environment, significantly enhancing the robustness and practicality of the system. Therefore, the application not only has significant advantages in the field of military and security, but also is widely applicable to many fields such as smart city management, disaster rescue communication, and environmental monitoring, showing great market demand and broad development prospects.

[0150] A computer-readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the steps of the reinforcement learning-based UAV-assisted ISAC system physical layer security transmission method.

[0151] The conventional techniques and schemes not described in detail in the above embodiments are well known in the art, and therefore will not be described in detail here. The above embodiments and / or experimental examples describe the preferred embodiments of the application in detail, however, the application is not limited to the specific details in the above embodiments, and within the scope of the technical concept of the application, the technical solutions of the application can be variously modified, and these simple modifications all belong to the protection scope of the application.

Claims

1. A method for secure physical layer transmission in a UAV-assisted ISAC system based on reinforcement learning, characterized in that, Includes the following steps: A UAV-assisted ISAC system transmission model is constructed, and a secure communication model and a sensing model are further constructed based on the UAV-assisted ISAC system transmission model. Based on the UAV-assisted ISAC system transmission model, secure communication model, and sensing model, under the constraints of total transmit power, communication user signal-to-interference-plus-noise ratio, and sensing target beammap gain performance, a system optimization model is established by jointly designing transmit beamforming, artificial noise, and UAV trajectory to maximize user and security rates; the optimization problem is expressed as: The decision variable is the beamforming vector of the transmitted signal. R v [n] is the covariance matrix of artificial noise, q u [n] = (x u [n],y u [n]) represents the horizontal position of the UAV itself; For the worst-case safe rate, P max P(θ) represents the maximum transmit power of the UAV. m [n])=a H (θ m [n])R x [n]a(θ m [n]) is the perception model, θ m [n] is the departure angle where the error exists, R x [n] is the covariance matrix of the transmitted signal; a(θ) m [n]) is the array steering vector, representing the steering vector of the m-th perceived target; Γ m For the lowest beammap gain, Γ c The lowest possible signal-to-interference-plus-noise ratio (SIR) for communication; h is the signal-to-interference-plus-noise ratio received at the legitimate user k. k [n] represents the channel vector from the UAV to the legitimate user k within time slot n, Δh k [n] represents the channel estimation error; V max The maximum flight speed of a UAV within a time slot. The initial position set for the UAV; x min x max y min y max Indicates the boundary of the region, (x u [n],y u [n],H) represents the time-varying position of the UAV in time slot n; By integrating the generative diffusion model into the Actor network of the TD3 algorithm, the GDMTD3 algorithm is designed to solve the optimization problem and generate the optimal action based on the current environmental state.

2. The physical layer secure transmission method for UAV-assisted ISAC system based on reinforcement learning according to claim 1, characterized in that, Constructing a UAV-assisted ISAC system transmission model, and further constructing a secure communication model and a sensing model based on the UAV-assisted ISAC system transmission model, including the following steps: Construct a UAV-assisted ISAC system transmission model: one UAV acts as a dual-function base station, communicating with legitimate user k; a sensing target m eavesdrops on the legitimate user's information; the location of the UAV changes within multiple time slots n; Establish a transmission signal model: In the nth time slot, the downlink transmission signal of the UAV is... In the formula This indicates a signal sent to the legitimate user k. The vector is the precoded vector of the transmitted signal. Represents embedded artificial noise. Let AN be the covariance matrix, and H represent the fixed flight altitude of the UAV; Establish an imperfect CSI model for communication users: Model the CSI of legitimate users and sensing targets, including legitimate user channels and eavesdropping channels, as well as model channel errors and angle errors.

3. The physical layer secure transmission method for UAV-assisted ISAC system based on reinforcement learning according to claim 2, characterized in that, The UAV is equipped with a uniform linear array of L antennas for transmitting beamforming; both the legitimate user and the sensing target are single antennas.

4. The physical layer secure transmission method for UAV-assisted ISAC system based on reinforcement learning according to claim 1, characterized in that, A secure communication model is constructed, including defining the secure rate for legitimate users; the expression for the secure rate in the worst case is given as follows: Where, [x] + =max{x,0}; The eavesdropping rate of the eavesdropper. It is the SINR of the m-th target eavesdropping on the k-th user; The transmission rate achievable by legitimate users; The system's perception model is constructed, including sensing targets by transmitting waveforms, and the perception performance is characterized using beammap gain, expressed as: P(θ m [n])=a H (i m [n])R x [n]a(θ m [n]) Where a(θ) m [n]) represents the guidance vector of the m-th perceived target.

5. The physical layer secure transmission method for UAV-assisted ISAC system based on reinforcement learning according to claim 1, integrating the generative diffusion model into the Actor network of the TD3 algorithm, and designing the GDMTD3 algorithm to solve the optimization problem, includes the following steps: The optimization problem is modeled as a Markov decision process, and the state space is... Action space reward function Define; By integrating GDM into the Actor network of the TD3 algorithm, complex state features are captured, and the optimal action is generated based on the current environmental state.

6. The physical layer secure transmission method for UAV-assisted ISAC system based on reinforcement learning according to claim 5, characterized in that, The state space definition The state of time slot n consists of the location of the UAV, the location information of the legitimate ground users and the sensing targets, and the channel state information; Where q u [n] is the position of the UAV, u k [n]、u m [n] represents the location information of the legitimate ground users and the sensing targets, respectively, h k [n]、g m [n] are the channel vectors from the UAV to the legitimate user and the sensing target, respectively; The action space Defined as the actions within time slot n, including the transmitted beamforming vector, the artificial noise covariance matrix, and the UAV flight angle and distance; a[n] = {w1[n],...w K [n],R v [n],φ[n],dis[n]}, where w k [n] is the transmitted signal beamforming vector, R v [n] is the covariance matrix of the artificial noise, and φ[n] and dis[n] represent the direction and distance of the UAV displacement, respectively; The reward function In the nth time slot, the instant reward is defined as: Where r p [n] represents the penalty imposed when a UAV flies out of the boundary, and ω is the penalty factor.

7. The physical layer secure transmission method for UAV-assisted ISAC system based on reinforcement learning according to claim 5 or 6, characterized in that, The optimization problem is modeled as a Markov decision process, including the following steps: A UAV is an agent; during each training time period, the agent first obtains the state s[n] from the environment; Then select strategy a[n] from the action space; Next, the environment updates its current state to s[n+1] and receives the corresponding reward r[n]. The empirical tuples s[n], a[n], r[n], s[n+1] are then stored in the replay memory buffer. middle.

8. The physical layer secure transmission method for UAV-assisted ISAC system based on reinforcement learning according to claim 1, characterized in that, The design of the GDMTD3 algorithm also includes the use of a parameterized model. To approximate the conditional distribution The parameterized model is represented as follows: in, The mean is denoted as g, and the condition information is denoted as g. The predetermined variance factor is denoted as: α represents all steps r≤t. r Cumulative product, where α t =1-β t ; The average value of the reverse process is calculated as follows: in, This is the original data; t is the number of diffusion steps. Let I be the distribution at step t, and β be the standard Gaussian noise. t Variance factor; Includes the following steps: For raw data The estimate is expressed as: in, It is a deep neural network; Based on the conditional information g, noise is denoised, and then the mean is indirectly approximated, expressed as: Tracking from arrive The reverse transformation establishes the generative distribution. as follows: in, Represents a standard normal distribution; Indicates that in the known Inverse denoising under certain conditions Conditional distribution; When generating distribution Once successfully trained, continue to obtain samples from the above formula.

9. The physical layer secure transmission method for UAV-assisted ISAC system based on reinforcement learning according to claim 8, characterized in that, The design of the GDMTD3 algorithm also includes a reparameterization process that is beneficial for sampling, as shown below: Where s represents the current state of the environment in DRL, As a condition variable in a parameterized function; ⊙ is the operator for the Hadamard product. This represents standard Gaussian noise.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the physical layer secure transmission method for a UAV-assisted ISAC system based on reinforcement learning as described in any one of claims 1 to 9.

Images