Multi-user energy collection resource allocation method based on unmanned aerial vehicle interference assistance
Through the D3QN algorithm and multi-agent framework, the drone interference and sub-user transmission power is optimized, and the problem of low system security rate when the location of the eavesdropper in the multi-user CR system is unknown, achieving more efficient communication security.
Patent Information
- Application Number
- CN202410928425.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-12
AI Technical Summary
In a multi-user CR system, where the eavesdropper location is unknown and mobile, how to optimize the drone interference power, interference trajectory, sub-user transmission power and transmission time slots to maximize the average confidentiality rate of the system, especially in TDMA Under the access method, the prior art is difficult to effectively solve this complex resource allocation problem.
Using a multi-agent framework based on D3QN algorithm, the drone and sub-user are regarded as independent agents to cooperate together. By designing state space, action space and reward functions, the drone trajectory and sub-user transmission power are optimized to realize the system optimal performance.
The system's worst-case average confidentiality rate was significantly improved. The simulation results showed that the sum of the average confidentiality rate of 10.9%, 56.8% and 63.9% was improved compared with other methods, ensuring communication security.
Smart Images

Figure CN118945686A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of energy harvesting resource allocation, and in particular relates to a multi-user energy harvesting resource allocation method based on unmanned aerial vehicle interference assistance. Background Art
[0002] Due to the existence of multiple users, in order to reduce interference between user communications, TDMA is generally used to transmit confidential information, and users communicate in turn according to the allocated time slots. When facing multiple eavesdroppers, the design and optimization of physical layer security (PLS) technology becomes more complicated, and existing research mainly focuses on the ideal situation where the eavesdropper's position is known, but in reality the eavesdropper's position is often unknown, its exact position is unknown, there are estimation errors, and the legitimate receiving node will also move. In this context, drones can be used as mobile friendly jammers to send artificial noise to interfere with potential eavesdroppers, considering the system average confidentiality rate in the worst case of the secondary user confidentiality rate in each time slot. The resource allocation problem in the multi-user CR system is complex and changeable. How to jointly optimize the interference power, interference trajectory, secondary user's transmission power and transmission time slot of the drone under the TDMA access mode to maximize the average confidentiality rate in the worst case is the focus of this application. Summary of the invention
[0003] The purpose of the present invention is to provide a multi-user energy harvesting resource allocation method based on drone interference assistance. Based on the D3QN algorithm, a multi-agent framework is designed, and the drone and each secondary user are regarded as independent agents to work together to achieve the optimal performance of the system PLS. Compared with other traditional methods, the method of the present invention has the best system average confidentiality rate.
[0004] To achieve the above objectives, the technical solution of the present invention is: a multi-user energy harvesting resource allocation method based on drone interference assistance, based on the D3QN algorithm, a multi-agent framework is designed, the drone and each secondary user are regarded as independent agents to work together to realize multi-user energy harvesting resource allocation.
[0005] In one embodiment of the present invention, the method comprises the following steps:
[0006] S1. Construct a multi-user energy harvesting CR system model for drone-assisted interference;
[0007] S2, the resource allocation problem of establishing a multi-user energy harvesting CR system model with drone-assisted jamming;
[0008] S3. Resource allocation based on DRL algorithm in multi-user scenarios.
[0009] In one embodiment of the present invention, step S1 is specifically implemented as follows:
[0010] In a three-dimensional coordinate system, consider a drone J that ensures secure communication with a secondary user network in the presence of an eavesdropper network; the secondary user network consists of multiple secondary transmitters and a randomly moving secondary receiver, with ST m represents the mth secondary transmitter, and SR represents the randomly moving secondary receiver, where The eavesdropper network consists of multiple eavesdroppers, but the exact location of each eavesdropper is unknown. n represents the nth eavesdropper, where The horizontal coordinate of the ground node G is denoted as q G =(x G ,y G ), where G∈{ST m ,SR,PT,PR,E n}(ST m (Secondary Transmitter): A secondary transmitter may be an auxiliary transmitter used in wireless communication to support or enhance the communication link; SR (Secondary Receiver): A secondary receiver may be mobile or fixed and is used to receive signals from a secondary transmitter or other sources; PT (Primary Transmitter): A primary transmitter usually refers to a transmitter used in a primary communication link and has priority; PR (Primary Receiver): A primary receiver used to receive signals from a primary transmitter; En (Eavesdropper): An eavesdropper may be a node that attempts to illegally obtain communication content); The drone always maintains a fixed altitude H (m) and starts horizontal flight from an initial position q0. The drone's working time W = LT is divided into L equal-length time slots, where T is the length of a unit time slot. The drone changes its horizontal position to q when flying in the tth time slot. J [t]=[x(t),y(t)] T , the flight angle set of the drone is Ω = {deg t}, t=1,···,L; Considering the Underly spectrum access mode, ST m Access the primary user spectrum to communicate with the SR in TDMA mode; each node is equipped with a single antenna, and the secondary transmitter is equipped with a RF energy harvester;
[0011] The position of the UAV (Unmanned Aerial Vehicle) satisfies the constraints
[0012] ||q J [t+1]-q J [t]|| 2≤U 2 ,t=1,...,L-1
[0013] ||q J [1]-q0|| 2 ≤U 2
[0014] -π≤deg t ≤π,t=1,...,L-1
[0015] Where U = Tv max represents the maximum horizontal distance of the UAV at the maximum speed in a time slot, v max is the maximum speed of the drone;
[0016] The system adopts a quasi-static channel model based on transmission blocks, in which the channel state information remains fixed in the current transmission block and may change in the next transmission block; the different fading sub-channels of each link between ground nodes obey independent quasi-static Rayleigh distributions, and the ground node link PT→PR, ST m →SR,ST m →PR,PT→SR,ST m →E n ,PT→ST m The channel coefficients are expressed as f PR , f m ,i m ,i SR , e m,n ,h m It means that there is f PR , f m ,i m ,i SR , e m,n , d is substituted according to the distance between nodes, d e is the path attenuation exponent; the eavesdropper's position can be estimated approximately, where ||·|| is the Euclidean norm, and Respectively represent the estimated location and actual location of the nth eavesdropper; is a non-negative real number representing the distance threshold. If the true position of an eavesdropper differs from the estimated position by no more than Then the position of the eavesdropper belongs to the set
[0017] An air-to-ground LoS channel is constructed between the UAV and the ground node. It is assumed that the Doppler effect of the UAV's maneuverability is fully compensated, and the fading subchannel in each link between the UAV and the ground node adopts the free space path loss model. The links J→PR, J→SR, and J→E of the UAV to the ground node in the tth time slot are n The coefficient gains are
[0018]
[0019] Where ρ0 represents the channel power gain at the reference distance d0 = 1m, d J→SR , d J→E and d J→PR is the distance between corresponding nodes; q PR ,q SR Respectively represent the positions of the primary receiver PR and the secondary receiver SR;
[0020] According to the sequential access mechanism of TDMA channels, in L transmission blocks, each transmission block time slot of length T is divided into two stages: energy collection and wireless information transmission. That is, each transmission block is M+1 transmission time slots, where the energy collection time is α t T, α t represents the energy collection efficiency, that is, the proportion of energy that the system can collect from the environment or signal at time t; the wireless information transmission time is (τ1+τ2+···+τ M )T; In the tth time block, PT instantaneously updates its transmit power p according to the power control strategy PT ; In the wireless information transmission stage, the signal is transmitted to PR, and the secondary transmitter ST m Access the primary user spectrum τ in turn m T time, and transmit confidential information to SR. At the same time, drone J sends confidential information to eavesdroppers E1, E2, ···, E n Transmit interference signal, τ m Indicates the time ratio or period of the secondary transmitter (ST) accessing the primary user (PU), τ m T represents the total time that the secondary transmitter accesses the primary user spectrum within the total time T; in each transmission block two-stage process, the time slot factor satisfies
[0021]
[0022] ST at the tth time block m and J in ST m The transmission power of the transmission time slot is respectively and p J,m Indicates that the RF signal of PT, ST mThe confidential signal of and the interference signal of J are independent cyclic symmetric complex Gaussian random variables, respectively. and X J,m Indicates that X is satisfied PT ~CN(0,p PT ), and X J,m ~CN(0,p J,m ), PR, SR and E n The received signals are
[0023]
[0024] Among them, the Gaussian noise signal received by PR, SR and E are all complex Gaussian random variables;
[0025] (1) Energy harvesting stage
[0026] During the energy harvesting phase, PT broadcasts a radio frequency signal X to PR. PT , each transmitter ST m Occupancy α t Continuously collect and store RF energy for T time; ST m The received RF power p STm,received Expressed as
[0027] p STm,received =α t η m p PT |h m | 2
[0028] Among them, η m For ST m Energy harvesting efficiency;
[0029] ST m The energy E collected at time t m [t] is expressed according to the nonlinear energy harvesting model as
[0030]
[0031] Ψ is a value between 0 and 1, Λ m Is with ST m Received RF power The related logic functions, parameters c and g are related to the actual EH circuit specifications, and B represents the maximum harvested power at the energy harvesting receiver when the EH reaches saturation;
[0032] (2) Maximum Transmit Power Constraint Analysis
[0033] In the wireless information transmission stage, ST mThe maximum transmit power constraint is
[0034]
[0035] Among them, p ST,max and p J,max ST m and the maximum transmit power of J;
[0036] (3) Signal-to-Interference-Noise Ratio Constraint Analysis
[0037] SR, PR and the nth eavesdropper E n The SINR of SR ,SINR PR , Indicates that the corresponding signal-to-interference-noise ratio threshold is λ SR ,λ PR , So SR, PR and the nth eavesdropper E n The SINR constraint condition is
[0038]
[0039] (4) Energy causal constraint analysis
[0040] The mth secondary transmitter ST m In the process of transmitting confidential information in the tth time block, the energy consumed by transmitting the information cannot exceed ST m The current available battery capacity, so the corresponding energy causal constraints are
[0041]
[0042] Among them, B m [t-1] is ST m The available battery capacity at the end of the t-1th time block, B ST,max For ST m The maximum available battery capacity.
[0043] In one embodiment of the present invention, in step S2, the resource allocation problem of the multi-user energy harvesting CR system model with drone-assisted interference, i.e., the problem of maximizing the confidentiality rate, is specifically as follows:
[0044] In each transmission time block, since the exact location of the eavesdropper cannot be known, the worst-case average confidentiality rate of the secondary user is considered as the physical layer security (PLS) performance indicator of the system, that is, only the link with the best eavesdropping channel quality in the t-th time block is considered, so the m-th secondary transmitter ST in the t-th time block m The confidentiality rate Defined as
[0045]
[0046] in, ST m →SR’s achievable rate and ST m →E n The eavesdropping rate is expressed as
[0047]
[0048] Considering the worst-case secondary user confidentiality rate and maximizing the average confidentiality rate, the optimization problem P1 is expressed as
[0049]
[0050] Through the UAV trajectory and variable τ m , p J The resource allocation solves the optimization problem P1. The trajectory of the UAV satisfies the constraints C1 to C3. Find the ST that satisfies the constraint C4. m The optimal transmission time slot τ m ,pass Calculate the energy harvesting time slot α and find the secondary user ST that meets the constraints C5 to C8 m and the optimal power p of UAV J STm and p J , to maximize the system's average confidentiality rate and ensure the secure transmission of secondary users' confidential information.
[0051] In one embodiment of the present invention, step S3 includes:
[0052] S31. Markov decision process:
[0053] When modeling the DRL agent and environment, the model is mapped to the interaction environment of MDP (abbreviation for "Markov Decision Process") and the state space is designed. Action Space And the reward function Three major mechanisms; among them, ST on the t-th time block m Transmit power Transmission time slot τ m and the UAV J transmission power p J,m As an agent, the trajectory of drone J is an agent, so the number of agents is M+1, and then the energy harvesting CR system environment, MDP and quintuple are modeled In the tth time block, Each agent obtains its own local observation Execute actions according to policy π and get rewarded Finally, according to the state transition probability, the current environment state s t Transition to next state s t+1 , and this completes the dynamic interaction.
[0054] In one embodiment of the present invention, the MDP is specifically implemented as follows:
[0055] (1) State space
[0056] For the UAV interference-assisted energy harvesting CR system in multi-user scenarios, the selected state includes the current location information of the UAV, the channel coefficient information between nodes, the signal-to-interference-noise ratio information, and the ST information of all secondary transmitters. m The battery capacity information of the environment can provide sufficient reference for the agent based on this state; the local observation value of the kth agent in the tth time block is
[0057]
[0058] in,
[0059]
[0060] Therefore, local observation The number of state variables is 2MN+4M+N+6; so all agents locally observe The environmental state of the composition t Expressed as
[0061]
[0062] S t The state number variable is (M+1)(2MN+4M+N+6);
[0063] (2) Action Space
[0064] In the resource allocation scenario of the UAV jammer-assisted CR system, the secondary transmitter ST m Need to m and Optimization, drone J needs to adjust the flight angle deg t and p J,m Optimize to make the system performance the best; therefore, the action of the drone trajectory agent is Indicates that the other m-th agents are Indicates that the optimization variable p J,m , τ m and degt The values are discretized into w1, w2, w3, and w4 levels respectively;
[0065]
[0066]
[0067] Among them, τ max For ST m Maximum transmission slot factor, deg min and deg max The maximum and minimum values of the UAV flight angle;
[0068] Each action is discretized, and the action space The size of is w1w2w3(2w4+1);
[0069] (3) Reward Function
[0070] In the UAV jamming-assisted CR system, the goal is to maximize the average confidentiality rate of the system. Each agent needs a good reward function to learn an effective optimal strategy that can solve the optimization problem so that the PLS performance of the system can converge. The main part of the sub-transmitter reward function is composed of the confidentiality rate The rest is composed of the signal to noise ratio to guide the agent to learn in the direction of higher SINR. The drone reward function adds an additional drone trajectory penalty term to guide the drone to fly within the specified range. Therefore, the transmitter ST m The reward function of drone J is defined as
[0071]
[0072] in,
[0073]
[0074] Among them, z2 and z3 are the maximum flight distances of the drone on the x-axis and y-axis respectively;
[0075] In DRL, the reward r[t] at a single time step constitutes the long-term return of the agent, so the final optimization problem is expressed as
[0076]
[0077] Among them, the discount factor γ∈(0,1) allows decision makers to adjust the importance of current and future rewards;
[0078] (4) State transition probability
[0079] The resource allocation problem in problem P1 is modeled as an MDP. The state transition probabilities involved in the MDP Experience data is accumulated through continuous interaction between the intelligent agent and the environment, and calculations are continuously iterated based on the Bellman equation.
[0080] In one embodiment of the present invention, step S3 further includes:
[0081] In the drone-assisted CR system, the optimization problem goal is to maximize the system's average confidentiality rate while meeting the primary user's QoS requirements. The combination of Double DQN and Dueling DQN, namely D3QN, is used as the training and target network, and the Prioritized Experience Replay PER mechanism is used, and the soft update method is set as the update rule. D3QN solves the problem of overestimation of Q value by using two network components to select actions and evaluate the value of the action, and separates two paths in the network architecture to calculate the state value function and the advantage function, respectively, and then combines them to estimate the Q value of each action, so that D3QN can more effectively learn more important actions in different states.
[0082] Let θ k and are the parameters of the decision network and target network of the kth agent, respectively. The weight parameters of the two branches in the Dueling network are w V and w A The output state value function and advantage function are represented by V(s t ;θ,w V ) and A(s t ,a t ;θ,w A ) indicates that the Q values output by the decision network and the target network are
[0083]
[0084] in, |A| is the size of the action space;
[0085] The PER mechanism based on TD error is used in the sampling process, and its error e is defined as
[0086] e k =r k [t]+κQ(s t+1 ,a′;θ k - )-Q(s t ,a t θ k )
[0087] Among them, κ∈(0,1) is the discount factor of the error;
[0088] In the PER mechanism, data with higher priority P(i) will be drawn from the experience pool D k The higher the probability of being selected as a training sample; the sampling priority P(i) is defined as
[0089]
[0090] Among them, φ is a constant between 0 and 1, which determines the influence of priority, u i is the priority of the i-th experience;
[0091] B extracted by the kth agent k The loss function L(θ k )for
[0092]
[0093] Decision network parameters θ k Minimize the loss function L(θ) by using the gradient descent algorithm k ) update, after a certain time step, the target network parameters Update according to the soft update method.
[0094] In one embodiment of the present invention, the method uses algorithm time complexity to evaluate efficiency.
[0095] The present invention also provides a multi-user energy harvesting resource allocation system based on drone interference assistance, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0096] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0097] Compared with the prior art, the present invention has the following beneficial effects: the present invention studies the problem of jointly optimizing the transmission time slot and transmission power of the secondary transmitter and maximizing the average confidentiality rate of the system by the UAV interference-assisted energy harvesting CR system under the conditions of meeting the QoS requirements of the primary user, the limited battery capacity of the secondary transmitter, the maximum transmission power constraint and the error of the eavesdropper's position. In order to solve this multi-constrained non-convex optimization problem, the present invention adopts the PER mechanism D3QN multi-agent resource allocation method, models each secondary user and drone as a multi-agent, and designs the relevant state space, action space and reward function. The simulation results show that the performance of the method of the present invention is better than other methods, and it can effectively improve the average confidentiality rate of the system in the worst case. Compared with the D3QN method without the PER mechanism, the proposed resource allocation method increases the sum of the average confidentiality rates of the system during the working time by 10.9%, 56.8% compared with the MADDPG method, and 63.9% compared with the DDQN. BRIEF DESCRIPTION OF THE DRAWINGS
[0098] Figure 1 This is the UAV interference-assisted multi-user energy harvesting CR system model of the present invention.
[0099] Figure 2 This is the interaction process between the intelligent agent of the present invention and the environment.
[0100] Figure 3 It is the D3QN algorithm framework of the present invention.
[0101] Figure 4 UAV jamming trajectories for different resource allocation methods.
[0102] Figure 5 is the average confidentiality rate change during working hours.
[0103] Figure 6 is the sum of the average confidentiality rates as p J,max Change curve.
[0104] Figure 7 is the sum of the average confidentiality rates as p ST,max Change curve.
[0105] Figure 8 Interference trajectories of drones under different primary user QoS requirements.
[0106] Fig. 9 The average confidentiality rate change during working hours for different primary user QoS requirements.
[0107] Fig.10 UAV interference trajectories with different estimation errors.
[0108] Fig.11The average confidentiality rate changes during working hours for different estimation errors.
[0109] Fig.12 It is the relationship between the sum of average confidentiality rates and ST maximum battery capacity.
[0110] Fig.13 This is the curve of average confidentiality rate changing with the number of iterations. DETAILED DESCRIPTION
[0111] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.
[0112] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0113] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0114] The present invention provides a multi-user energy harvesting resource allocation method based on UAV interference assistance. Based on the D3QN algorithm, a multi-agent framework is designed to regard the UAV and each secondary user as independent agents to work together to achieve multi-user energy harvesting resource allocation. The D3QN algorithm can effectively process high-dimensional state and action spaces, and can learn stable and optimal strategies in an environment with uncertainty. Combined with the PER mechanism, it can better utilize important experience and improve learning efficiency. Simulation results show that the proposed resource allocation method achieves the best system average confidentiality rate compared with other traditional methods. The contributions of this invention are as follows:
[0115] (1) In the presence of multiple eavesdroppers, inaccurate location information, and mobile receiving nodes, the present invention studies the PLS performance enhancement problem of a multi-user CR system with energy harvesting function. The average confidentiality rate of the secondary user in the worst case is taken as the performance indicator to maximize the sum of the long-term average confidentiality rate of the system.
[0116] (2) For the complex scenarios studied, the D3QN algorithm is adopted, which can more effectively learn more important actions under different states, and is combined with the PER mechanism to improve the training efficiency. The complex optimization problem is established as an MDP. It is proved that the proposed resource allocation method under different constraints can effectively improve the average confidentiality rate of the system compared with other methods, and ensure the communication security of multiple users facing multiple eavesdroppers whose exact locations are unknown.
[0117] like Figure 1 As shown, this embodiment provides a multi-user energy harvesting resource allocation method based on drone interference assistance, including:
[0118] 1. Construct a multi-user energy harvesting CR system model for drone-assisted jamming
[0119] Building an energy harvesting CR system such as Figure 1 As shown, in this three-dimensional coordinate system, the present invention considers a UAV jammer J to ensure the secure communication of a secondary user network in the presence of an eavesdropper network. The secondary user network consists of multiple secondary transmitters and a randomly moving secondary receiver, with ST m represents the mth secondary transmitter, and SR represents the randomly moving secondary receiver, where The eavesdropper network consists of multiple eavesdroppers, but the exact location of each eavesdropper is unknown. n represents the nth eavesdropper, where The horizontal coordinate of the ground node G is denoted as q G =(x G ,y G ), where G∈{ST m ,SR,PT,PR,E n The drone always maintains a fixed height H(m) and starts to fly horizontally from the initial position q0. The drone's working time W=LT is divided into L equal-length time slots, T is the unit time slot length, and the drone's horizontal position changes to q when flying in the tth time slot. J [t]=[x(t),y(t)] T , the flight angle set of the drone is Ω = {deg t}, t=1,···,L. Considering the Underly spectrum access mode, ST m The primary user spectrum is accessed in TDMA mode to communicate with the SR. Each of the above nodes is equipped with a single antenna, and the secondary transmitter is equipped with a RF energy harvester.
[0120] The position of the UAV (Unmanned Aerial Vehicle) satisfies the constraints
[0121] ||q J [t+1]-qJ [t]|| 2 ≤U 2 ,t=1,...,L-1 (1)
[0122] ||q J [1]-q0|| 2 ≤U 2 (2)
[0123] -π≤deg t ≤π,t=1,...,L-1 (3)
[0124] Where U = Tv max Indicates the maximum horizontal distance that the drone can travel at maximum speed (m / s) in a time slot. max is the maximum speed of the drone.
[0125] The system adopts a quasi-static channel model based on transmission blocks, where the channel state information remains fixed in the current transmission block and may change in the next transmission block. The different fading subchannels of each link between ground nodes obey independent quasi-static Rayleigh distributions. The ground node link PT→PR, ST m →SR,ST m →PR,PT→SR,ST m →E n ,PT→ST m The channel coefficients are expressed as f PR , f m ,i m ,i SR , e m,n ,h m It means that there is f PR , f m ,i m ,i SR , e m,n , ), d is substituted according to the distance between nodes, d e is the path attenuation exponent. The eavesdropper’s position can be estimated approximately by where ||·|| is the Euclidean norm, and represent the estimated location and actual location of the nth eavesdropper respectively.
[0126] An air-to-ground LoS channel is constructed between the drone and the ground node. It is assumed that the Doppler effect of the drone's mobility is fully compensated, and the fading subchannel in each link between the drone and the ground node adopts the free space path loss model. The link J→PR, J→SR, and J→E from the drone to the ground node in the tth time slot are n The coefficient gains are
[0127]
[0128] Where ρ0 represents the channel power gain at the reference distance d0 = 1m, d J→SR , d J→E and d J→PR is the distance between corresponding nodes.
[0129] According to the sequential access mechanism of TDMA channels, in L transmission blocks, each transmission block time slot of length T is divided into two stages: energy collection and wireless information transmission. That is, each transmission block is M+1 transmission time slots, where the energy collection time is α t T, the wireless information transmission time is (τ1+τ2+···+τ M )T. In the tth time block, PT instantaneously updates its transmit power p according to the power control strategy PT In the wireless information transmission stage, the signal is transmitted to PR, and the secondary transmitter ST m Access the primary user spectrum τ in turn m T time, and transmit confidential information to SR. At the same time, drone J sends confidential information to eavesdroppers E1, E2, ···, E n In the above two-stage process of each transmission block, the time slot factor satisfies
[0130]
[0131] ST at the tth time block m and J in ST m The transmission power of the transmission time slot is respectively and p J,m Indicates that the RF signal of PT, ST m The confidential signal of and the interference signal of J are independent cyclic symmetric complex Gaussian random variables, respectively. and X J,m Indicates that X is satisfied PT ~CN(0,p PT ), and X J,m ~CN(0,p J,m ), PR, SR and E n The received signals are
[0132]
[0133] Among them, PR, SR and E receive Gaussian noise signals are all complex Gaussian random variables.
[0134] (1) Energy harvesting stage
[0135] During the energy harvesting phase, PT broadcasts a radio frequency signal X to PR. PT , each transmitter ST m Occupancy α t Continuously collect and store RF energy for T time. ST m Received RF power It can be expressed as
[0136]
[0137] Among them, η m For ST m energy harvesting efficiency.
[0138] ST m The energy E collected at time t m [t] can be expressed as follows according to the nonlinear energy harvesting model:
[0139]
[0140] Λ m Is with ST m Received RF power The related logic function, parameters c and g are related to the actual EH circuit specifications, and B represents the maximum harvested power at the energy harvesting receiver when the EH reaches saturation.
[0141] (2) Maximum Transmit Power Constraint Analysis
[0142] In the wireless information transmission stage, ST m The maximum transmit power constraint is
[0143]
[0144] Among them, p ST,max and p J,max ST m and the maximum transmit power of J.
[0145] (3) Signal-to-Interference-Noise Ratio Constraint Analysis
[0146] SR, PR and the nth eavesdropper E n The SINR of SR ,SINR PR , Indicates that the corresponding signal-to-interference-noise ratio threshold is λ SR ,λ PR , So SR, PR and the nth eavesdropper E n The SINR constraint condition is
[0147]
[0148] (4) Energy causal constraint analysis
[0149] The mth secondary transmitter ST m In the process of transmitting confidential information in the tth time block, the energy consumed by transmitting the information cannot exceed ST m The current available battery capacity, so the corresponding energy causal constraints are
[0150]
[0151] Among them, B m [t-1] is ST m The available battery capacity at the end of the t-1th time block, B ST,max For ST m The maximum available battery capacity.
[0152] 2. Analysis of resource allocation issues
[0153] 2.1 Analysis of the problem of maximizing confidentiality rate
[0154] In each transmission time block, since the exact location of the eavesdropper cannot be known, the present invention considers the worst-case average confidentiality rate of the secondary user as the PLS performance indicator of the system, that is, only the link with the best eavesdropping channel quality in the tth time block is considered. Therefore, the mth secondary transmitter ST in the tth time block m The confidentiality rate Can be defined as
[0155]
[0156] in, ST m →SR’s achievable rate and ST m →E n The eavesdropping rate is expressed as
[0157]
[0158] Considering the worst-case secondary user confidentiality rate and maximizing the average confidentiality rate, the optimization problem P1 can be expressed as
[0159]
[0160] The present invention aims to achieve reasonable UAV trajectory and variable τ m , p J The resource allocation solves the optimization problem P1. The trajectory of the UAV satisfies the constraints C1 to C3. Find the ST that satisfies the constraint C4. m The optimal transmission time slot τ m ,pass Calculate the energy harvesting time slot α and find the secondary user ST that meets the constraints C5 to C8 m and the optimal power of the UAV jammer J and p J , to maximize the system's average confidentiality rate and ensure the secure transmission of secondary users' confidential information.
[0161] 3. Design of resource allocation method based on DRL algorithm in multi-user scenario
[0162] 3.1 Markov Decision Process
[0163] When modeling DRL agents and environments, the models are mapped to the interaction environment of MDPs and the state space is designed. Action Space And the reward function Three major mechanisms. Among them, ST on the tth time block m Transmit power Transmission time slot τ m and the jammer J transmission power p J,m As an agent, J trajectory as an agent, so the number of agents is M+1, and then the energy harvesting CR system environment is modeled, such as Figure 2 As shown. System MDP and quintuple In the tth time block, Each agent obtains its own local observation Perform actions according to the policy π and get rewarded Finally, according to the state transition probability, from the current environment state s t Transition to next state s t+1 , so far a dynamic interaction ends. The key elements of MDP are described as follows.
[0164] (1) State space
[0165] In the UAV interference-assisted energy harvesting CR system in multi-user scenarios, in order to make the learning efficiency of the intelligent agent higher, the state selected by the present invention includes the current position information of the UAV, the channel coefficient information between nodes, the signal-to-interference-noise ratio information and all secondary transmitters ST m The battery capacity information of the environment can provide sufficient reference for the agent based on this environment state. The local observation value of the kth agent in the tth time block is
[0166]
[0167] in,
[0168]
[0169] Therefore, local observation The number of state variables is 2MN+4M+N+6. Therefore, all agents locally observe The environmental state of the composition t Expressed as
[0170]
[0171] S t The state number variable is (M+1)(2MN+4M+N+6).
[0172] (2) Action Space
[0173] In the resource allocation scenario of the UAV jammer-assisted CR system, the secondary transmitter ST m Need to m and Optimization, UAV jammer J needs to adjust the flight angle deg t and p J,m Optimize to make the system performance the best. Therefore, the action of the drone trajectory agent is Indicates that the other m-th agents are In order to reduce the amount of calculation, the present invention optimizes the variables p J,m , τ m and deg t The values are discretized into w1, w2, w3, and w4 levels respectively.
[0174]
[0175]
[0176] Among them, τ max For ST m Maximum transmission slot factor, deg min and deg max The maximum and minimum values of the drone’s flight angle.
[0177] After each action is discretized, the size of the action space A is w1w2w3(2w4+1).
[0178] (3) Reward Function
[0179] In the UAV jamming-assisted CR system, the goal is to maximize the average confidentiality rate of the system. Each agent needs a good reward function to learn an effective optimal strategy that can solve the optimization problem so that the PLS performance of the system can converge. The main part of the sub-transmitter reward function is composed of the confidentiality rate The rest of the components are signal-to-interference-noise ratio to guide the agent to learn in the direction of higher SINR. The drone reward function adds an additional drone trajectory penalty term to guide the drone to fly within the specified range. Therefore, the transmitter ST m The reward function of drone J can be defined as
[0180]
[0181] in,
[0182]
[0183] Among them, z2 and z3 are the maximum flight distances of the drone on the x-axis and y-axis respectively.
[0184] Therefore, in DRL, the reward r[t] of a single time step constitutes the long-term return of the agent, so the last optimization problem of the present invention is expressed as
[0185]
[0186] Here, the discount factor γ∈(0,1) allows the decision maker to adjust the importance of current and future rewards.
[0187] (4) State transition probability
[0188] The resource allocation problem in problem (17) is modeled as an MDP. The state transition probability P involved in the MDP accumulates empirical data through continuous interaction between the agent and the environment, and is continuously iterated and calculated according to the Bellman equation.
[0189] Bellman equation
[0190]
[0191] V * (s) is the maximum expected reward that can be obtained by starting from state s and following the optimal policy.
[0192] A is the set of all actions that can be taken in state s.
[0193] R(s,a) is the immediate reward of taking action a to move from state s to a certain state.
[0194] γ is the discount factor, which is between 0 and 1 and indicates the value of future returns relative to current returns.
[0195] P(s′|s,a) is the probability of transitioning from state s to state s′ by taking action a.
[0196] S is the set of all possible states.
[0197] 3.2 Resource Allocation Algorithm Framework Design
[0198] In the drone-assisted CR system, the optimization problem involves coordinating multiple resources such as interference power, secondary user transmission power, secondary user transmission time slot and drone trajectory in a dynamic environment to maximize the average confidentiality rate of the system while meeting the QoS requirements of the primary user. The present invention adopts a combination of Double DQN and Dueling DQN (Dueling Double DeepQ-network, D3QN) as our training and target network, uses the Prioritized Experience Replay (PER) mechanism to improve sample utilization and further reduce decision time, and sets a soft update method as the update rule. D3QN solves the problem of overestimation of Q value by forming two networks for selecting actions and evaluating the value of the action respectively, and divides two paths in the network architecture for calculating the state value function and the advantage function respectively, and then combines them to estimate the Q value of each action, so that D3QN can more effectively learn more important actions under different states.
[0199] Let θ k and are the parameters of the decision network and target network of the kth agent, respectively. The weight parameters of the two branches in the Dueling network are w V and w A The output state value function and advantage function are represented by V(s t ;θ,w V ) and A(s t ,a t ;θ,w A ), so the Q values output by the decision network and the target network are
[0200]
[0201] in, |A| is the size of the action space.
[0202] The PER mechanism based on TD error is used in the sampling process, and its error e is defined as
[0203]
[0204] Among them, κ∈(0,1) is the discount factor of the error.
[0205] In the PER mechanism, data with higher priority P(i) will be drawn from the experience pool D k The higher the probability of being selected as a training sample. The sampling priority P(i) is defined as
[0206]
[0207] Among them, φ is a constant between 0 and 1, which determines the influence of priority, u i is the priority of the ith experience.
[0208] B extracted by the kth agent k The loss function L(θ k )for
[0209]
[0210] Decision network parameters θ k Minimize the loss function L(θ) by using the gradient descent algorithm k ) update, after a certain time step, the target network parameters Update according to the soft update method. The specific process of the algorithm is shown in Algorithm Table 1.
[0211] Table 1 D3QN resource allocation algorithm
[0212]
[0213]
[0214] 3.3 Algorithm time complexity analysis
[0215] Algorithm complexity is one of the important criteria for evaluating algorithm efficiency. By describing how the running time of an algorithm grows with the growth of the input data dimension, an upper bound estimate of the time growth trend in the worst case or average case can be calculated.
[0216] Denoting the complexity of the D3QN execution phase as O(Γ), for a network with an input vector dimension of I and an output dimension of O1, the forward propagation process is as follows Figure 3 As shown, the first four fully connected layers, the final output layer is connected to the softmax layer, and the i-th fully connected layer uses N i Therefore, O(Γ) is calculated based on the principle of calculation.
[0217]
[0218] Among them, it can be seen that due to the Dueling DQN network architecture, there will be three network layers fully connected to the second fully connected layer, and the output part of the softmax layer is equal to the output dimension, so O(softmax) = O1
[0219] Prioritize the experience playback process, each step requires D in the experience pool k After sorting, sample a certain batch of sample data It is used to minimize formula (36) to update network parameters, where B is the update batch dimension. When a new data is stored in the experience pool, it will be assigned a priority, so the complexity of calculating a data priority is equal to the complexity of a network forward propagation, that is, O(Γ). The complexity of extracting B times of experience is
[0220]
[0221] Among them, |D k |For the The experience pool capacity of an agent.
[0222] In one round, there are L steps, and the target network updates its parameters every N′ steps, where L is divisible by N′. Therefore, the update complexity of the target network in one round is O(Π)
[0223]
[0224] So the complexity of the training phase is
[0225]
[0226] 4. Simulation results and analysis
[0227] The present invention aims at a multi-user energy harvesting CR system assisted by UAV interference, taking into account the random movement of SR and the inaccurate position of E, and verifies the effectiveness of the resource allocation method of the present invention in improving the PLS performance of the system through simulation. The main parameters of the environment are shown in Table 2. The present invention also considers the premise that the initial position of the interference is close to SR, and assumes that the initial position of the UAV is q[0]=[0,200] T , the initial positions of PR and SR and the estimated position of E are q PR =(500,200,0),q SR =(150,200,0),q E1 =(150,400,0),q E2 =(300,100,0) and q E3 =(300,350,0), the drone's motion direction is [-π,π]. The drone's flight altitude is H = 100m, the time slot length is 1s, ρ0 = -40dB, Path Loss Index Position estimation error Since the effectiveness of reinforcement learning in resource optimization has been demonstrated in Chapter 3, this paper only considers the working range of 700×700m 2The experiment used Python 3.6 to develop the DRL environment, built and executed the deep neural network through Pytorch 1.6, and used the Adam optimizer for gradient descent. In addition to comparing D3QN without PER, the present invention also compared with the following benchmark solutions:
[0228] (1) Resource allocation based on MADDPG: The DDPG algorithm is extended to multi-agent systems, allowing each agent to utilize global information during training but only rely on its own local observation information during actual execution. This approach is suitable for dealing with complex interaction problems in multi-agent environments.
[0229] (2) Resource allocation based on DDQN: The DQN algorithm is improved by using two networks to reduce the deviation of value estimation, thereby improving the stability and accuracy of the algorithm.
[0230] Table 2 Simulation parameters
[0231]
[0232] Figure 4 and Figure 5 They are the interference trajectory of the friendly jammer J under different algorithms and the change of the average confidentiality rate during working time. Under the premise that the moving SR is closer to the initial position of J, in the face of multiple eavesdroppers with uncertain positions, in order to ensure the long-term maximization of the worst-case confidentiality rate of each time block of SR, as follows Figure 4 As shown in , since J interference will also affect SR, all trajectories will move away from SR at the beginning, looking for their own optimal position to effectively interfere with the eavesdropping link while trying to protect the channel quality of the legitimate receiver. However, the optimal trajectory cannot be determined based on the trajectory alone. Figure 5 As shown in the figure, all methods effectively improve the confidentiality rate of the system, and the sum of the confidentiality rates of the method proposed in the present invention during the working time is the largest, which proves that the trajectory of this method is optimal. It can be seen from the change of the confidentiality rate curve that the quality of the eavesdropper's channel is better than the legitimate channel of the secondary user at the beginning, so the initial confidentiality rate will not increase. As J's trajectory is optimized and the interference noise is emitted to effectively interfere with the eavesdropper, the confidentiality rate will also increase. Limited by energy constraints, only reasonable resource allocation can be adopted to obtain the maximum average confidentiality rate in the end, so the method proposed in the present invention has not always been in the leading position.
[0233] Figure 6 and Figure 7 The maximum jamming power p emitted by the UAV jammer J is studied. J,max and secondary user ST m The maximum transmission power p ST,max The impact of changes on the sum of the average confidentiality rates. Figure 6 It can be seen that with the increase of interference power, the average confidentiality rate of each method increases to varying degrees, among which the method proposed by the present invention increases the fastest, and the increase begins to slow down when the maximum power reaches 2W. Excessive interference power cannot guarantee a stable increase in the confidentiality rate of the system, because the communication QoS of the primary user must also be guaranteed at the same time, so it is important to select an appropriate interference power. Figure 7 It can be seen that the average confidentiality rate of the system will also increase with the increase of the maximum power of the secondary user, but when the maximum power is 2.5W, each method converges at different levels, among which the convergence value of the method proposed in the present invention is the highest. The main reason why the increase of the maximum power of the secondary user converges is related to its own limited energy. Insufficient energy cannot support excessive transmission power. This also proves the importance of energy harvesting in the system and the resource allocation of the method proposed in the present invention is better than other methods, which can effectively improve the average confidentiality rate of the system.
[0234] Figure 8 and Fig. 9 They are respectively the trajectory of the UAV jammer and the change of the average confidentiality rate during working hours under different QoS requirements of the primary user. Figure 8 As shown in Figure 1, when the minimum QoS requirement of the primary user is higher, following the free space path loss channel model, the farther the jammer is from the primary user, the smaller the interference it causes to it, so different QoS requirements will correspond to different optimal trajectories. Fig. 9 It can be seen that the higher the QoS, the more it will affect the transmission strategy of the secondary user. The learned action strategy should avoid affecting the normal communication of the primary user. Therefore, the average confidentiality rate of the system will decrease as the QoS requirements of the primary user increase.
[0235] Fig.10 and Fig.11 Different estimation errors The impact on the jammer J and the average confidentiality rate of the system. Figure 4-10 It can be seen that the resource allocation method proposed in the present invention can still optimize a reasonable trajectory within a certain error, but when the error reaches 70m, the trajectory will be farther away from the actual eavesdropping location. Fig.11 It can be seen that the increase of error distance will also have a certain impact on performance, and the sum of the average confidentiality rates decreases as the error increases. Fig.10 and Fig.11 It can be seen that even when there is a large error in the eavesdropper's position, the method proposed in the present invention can well improve the average confidentiality rate of the system.
[0236] For energy harvesting, Fig.12The relationship between the maximum battery capacity of ST and the average confidentiality rate of the system was studied, and a comparison of the proposed algorithm without energy harvesting was added. It can be seen from the figure that each method reached different degrees of convergence when the battery capacity of ST was 120, and the convergence effect of the method proposed in the present invention was the best, because the convergence result of D3QN was close to that of the method proposed in the present invention. Therefore, as the battery capacity increases, the convergence value of the sum of the average confidentiality rates of the system is close to that of the method proposed in the present invention. Since the energy required during working time has a maximum value, as the battery capacity increases, the effect of energy harvesting on the system is also weakened. The method without energy harvesting converges to the same level as the proposed method when the battery capacity is large enough. Therefore, the average confidentiality rate of the system will not increase with the increase of battery capacity. When it exceeds the energy required during working time, even if it increases further, it will only be excess energy that cannot be consumed in time.
[0237] Fig.13 The variation of the average confidentiality rate in the training phase with the number of training rounds was studied. As can be seen from the figure, as the number of iterations increases, all methods show an overall upward trend, and the method proposed in the present invention has the highest convergence value and a fast convergence speed compared with other methods, converging in about 200 rounds. The convergence result of the D3QN method without the PER mechanism is slightly worse than that of the method proposed in the present invention, but the convergence speed is much lower than that of the method proposed in the present invention, converging in about 400 rounds. The MADDPG method converges quickly, converging in about 150 rounds, but the convergence value is not ideal. The convergence speed of the DDQN method is similar to that of D3QN, but the convergence value is the worst among all methods.
[0238] In summary, analysis is carried out on different main user QoS requirements, different estimation errors, different maximum transmission powers and ST battery capacities, and simulation proves that the method proposed in the present invention can effectively solve the PLS enhancement problem in the UAV interference-assisted energy harvesting CR system, so that the average confidentiality rate of the system reaches the maximum value, ensuring the communication security of the system.
[0239] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0240] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0241] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0242] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0243] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
Claims
1. A multi-user energy harvesting resource allocation method based on drone interference assistance, characterized in that: Based on the D3QN algorithm, a multi-agent framework is designed, which regards the drone and each secondary user as independent agents to collaborate and realize multi-user energy harvesting resource allocation.
2. According to claim 1, a multi-user energy harvesting resource allocation method based on drone interference assistance is characterized in that: The method comprises the following steps: S1. Construct a multi-user energy harvesting CR system model for drone-assisted interference; S2, the resource allocation problem of establishing a multi-user energy harvesting CR system model with drone-assisted jamming; S3. Resource allocation based on DRL algorithm in multi-user scenarios.
3. The multi-user energy harvesting resource allocation method based on UAV interference assistance according to claim 2 is characterized in that: Step S1 is specifically implemented as follows: In a three-dimensional coordinate system, consider a drone J that ensures secure communication with a secondary user network in the presence of an eavesdropper network; the secondary user network consists of multiple secondary transmitters and a randomly moving secondary receiver, with ST m represents the mth secondary transmitter, and SR represents the randomly moving secondary receiver, where The eavesdropper network consists of multiple eavesdroppers, but the exact location of each eavesdropper is unknown. n represents the nth eavesdropper, where The horizontal coordinate of the ground node G is denoted as q G =(x G ,y G ), where G∈{ST m ,SR,PT,PR,E n }, PT is the main transmitter, PR is the main receiver, used to receive the signal of the main transmitter; the drone always maintains a fixed height H (m) and starts horizontal flight from the initial position q0, and the drone's working time W = LT is divided into L equal-length time slots, T is the unit time slot length, and the drone changes its horizontal position to q when flying in the tth time slot J [t]=[x(t),y(t)] T , the flight angle set of the drone is Ω = {deg t }, t=1,···,L; Considering the Underly spectrum access mode, ST m Access the primary user spectrum to communicate with the SR in TDMA mode; each node is equipped with a single antenna, and the secondary transmitter is equipped with a RF energy harvester; The position of the drone satisfies the constraints ||q J [t+1]-q J [t]|| 2 ≤U 2 ,t=1,...,L-1 ||q J [1]-q0|| 2 ≤U 2 -π≤deg t ≤π,t=1,...,L-1 Where U = Tv max represents the maximum horizontal distance of the UAV at the maximum speed in a time slot, v max is the maximum speed of the drone; The system adopts a quasi-static channel model based on transmission blocks, in which the channel state information remains fixed in the current transmission block and may change in the next transmission block; the different fading sub-channels of each link between ground nodes obey independent quasi-static Rayleigh distributions, and the ground node link PT→PR, ST m →SR,ST m →PR,PT→SR,ST m →E n ,PT→ST m The channel coefficients are expressed as f PR , f m ,i m ,i SR , e m,n ,h m It means that there is f PR , f m ,i m ,i SR , e m,n , d is substituted according to the distance between nodes, d e is the path attenuation exponent; the eavesdropper's position can be estimated approximately, where ||·|| is the Euclidean norm, and Respectively represent the estimated location and actual location of the nth eavesdropper; is a non-negative real number representing the distance threshold. If the true position of an eavesdropper differs from the estimated position by no more than Then the position of the eavesdropper belongs to the set An air-to-ground LoS channel is constructed between the UAV and the ground node. It is assumed that the Doppler effect of the UAV's maneuverability is fully compensated, and the fading subchannel in each link between the UAV and the ground node adopts the free space path loss model. The links J→PR, J→SR, and J→E of the UAV to the ground node in the tth time slot are n The coefficient gains are Where ρ0 represents the channel power gain at the reference distance d0 = 1m, d J→SR , d J→E and d J→PR is the distance between corresponding nodes; q PR ,q SR Respectively represent the positions of the primary receiver PR and the secondary receiver SR; According to the sequential access mechanism of TDMA channels, in L transmission blocks, each transmission block time slot of length T is divided into two stages: energy collection and wireless information transmission. That is, each transmission block is M+1 transmission time slots, where the energy collection time is α t T, α t represents the energy collection efficiency, that is, the proportion of energy that the system can collect from the environment or signal at time t; the wireless information transmission time is (τ1+τ2+···+τ M )T; In the tth time block, PT instantaneously updates its transmit power p according to the power control strategy PT ; In the wireless information transmission stage, the signal is transmitted to PR, and the secondary transmitter ST m Access the primary user spectrum τ in turn m T time, and transmit confidential information to SR. At the same time, drone J sends confidential information to eavesdroppers E1, E2, ···, E n Transmit interference signal, τ m Indicates the time ratio or period of ST access to PU; in each transmission block two-stage process, the time slot factor satisfies ST at the tth time block m and J in ST m The transmission power of the transmission time slot is respectively and p J,m Indicates that the RF signal of PT, ST m The confidential signal of and the interference signal of J are independent cyclic symmetric complex Gaussian random variables, respectively. and X J,m Indicates that X is satisfied PT ~CN(0,p PT ), and X J,m ~CN(0,p J,m ), PR, SR and E n The received signals are Among them, the Gaussian noise signal received by PR, SR and E are all complex Gaussian random variables; (1) Energy collection stage During the energy harvesting phase, PT broadcasts a radio frequency signal X to PR. PT , each transmitter ST m Occupancy α t T time to continuously collect and store RF energy; ST m Received RF power Expressed as Among them, η m For ST m Energy harvesting efficiency; ST m The energy E collected at time t m [t] is expressed according to the nonlinear energy harvesting model as Ψ is a value between 0 and 1, Λ m Is with ST m Received RF power The related logic functions, parameters c and g are related to the actual EH circuit specifications, and B represents the maximum harvested power at the energy harvesting receiver when the EH reaches saturation; (2) Maximum Transmit Power Constraint Analysis In the wireless information transmission stage, ST m The maximum transmit power constraint is Among them, p ST,max and p J,max ST m and the maximum transmit power of J; (3) Signal-to-Interference-Noise Ratio Constraint Analysis SR, PR and the nth eavesdropper E n The SINR of SR ,SINR PR , Indicates that the corresponding signal-to-interference-noise ratio threshold is λ SR ,λ PR , So SR, PR and the nth eavesdropper E n The SINR constraint is (4) Energy causal constraint analysis The mth secondary transmitter ST m In the process of transmitting confidential information in the tth time block, the energy consumed by transmitting the information cannot exceed ST m The current available battery capacity, so the corresponding energy causal constraints are Among them, B m [t-1] is ST m The available battery capacity at the end of the t-1th time block, B ST,max For ST m The maximum available battery capacity.
4. The multi-user energy harvesting resource allocation method based on UAV interference assistance according to claim 3 is characterized in that: In step S2, the resource allocation problem of the multi-user energy harvesting CR system model with drone-assisted interference, i.e., the problem of maximizing the confidentiality rate, is as follows: In each transmission time block, since the exact location of the eavesdropper cannot be known, the worst-case average confidentiality rate of the secondary user is considered as the physical layer security PLS performance indicator of the system, that is, only the link with the best eavesdropping channel quality in the t-th time block is considered, so the m-th secondary transmitter ST in the t-th time block m The confidentiality rate Defined as in, ST m →SR’s achievable rate and ST m →E n The eavesdropping rate is expressed as Considering the worst-case secondary user confidentiality rate and maximizing the average confidentiality rate, the optimization problem P1 is expressed as Through the UAV trajectory and variable τ m , p J The resource allocation solves the optimization problem P1. The trajectory of the UAV satisfies the constraints C1 to C3. Find the ST that satisfies the constraint C4. m The optimal transmission time slot τ m ,pass Calculate the energy harvesting time slot α and find the secondary user ST that meets the constraints C5 to C8 m and the optimal power of UAV J and p J , to maximize the system's average confidentiality rate and ensure the secure transmission of secondary users' confidential information.
5. The multi-user energy harvesting resource allocation method based on UAV interference assistance according to claim 4 is characterized in that: Step S3 includes: S31. Markov decision process: When modeling DRL agents and environments, the model is mapped to the interactive environment of Markov decision processes (MDPs) and the state space is designed. Action Space And the reward function Three major mechanisms; among them, ST on the t-th time block m Transmit power Transmission time slot τ m and the UAV J transmission power p J,m As an agent, the trajectory of drone J is an agent, so the number of agents is M+1, and then the energy harvesting CR system environment, MDP and quintuple are modeled is the state transition probability, γ is the discount factor, in the tth time block, Each agent obtains its own local observation Execute actions according to policy π And get reward t k Finally, according to the state transition probability, the current environment state s t Transition to next state s t+1 , and this completes the dynamic interaction.
6. The multi-user energy harvesting resource allocation method based on UAV interference assistance according to claim 5 is characterized in that: The MDP is specifically implemented as follows: (1) State space For the UAV interference-assisted energy harvesting CR system in multi-user scenarios, the selected state includes the current location information of the UAV, the channel coefficient information between nodes, the signal-to-interference-noise ratio information, and the ST information of all secondary transmitters. m The battery capacity information of the environment can provide sufficient reference for the agent based on this state; the local observation value of the kth agent in the tth time block is in, Therefore, local observation The number of state variables is 2MN+4M+N+6; so all agents locally observe The environmental state of the composition t Expressed as S t The state number variable is (M+1)(2MN+4M+N+6); (2) Action Space In the resource allocation scenario of the UAV jammer-assisted CR system, the secondary transmitter ST m Need to m and Optimization, drone J needs to adjust the flight angle deg t and p J,m Optimize to make the system performance the best; therefore, the action of the drone trajectory agent is Indicates that the other mth agents are Indicates that the optimization variable p J,m , τ m and deg t The values are discretized into w1, w2, w3, and w4 levels respectively; Among them, τ max For ST m Maximum transmission slot factor, deg min and deg max The maximum and minimum values of the UAV flight angle; Each action is discretized, and the action space The size of is w1w2w3(2w4+1); (3) Reward Function In the UAV jamming-assisted CR system, the goal is to maximize the average confidentiality rate of the system. Each agent needs a good reward function to learn an effective optimal strategy that can solve the optimization problem so that the PLS performance of the system can converge. The main part of the sub-transmitter reward function is composed of the confidentiality rate The rest is composed of the signal to noise ratio to guide the agent to learn in the direction of higher SINR. The drone reward function adds an additional drone trajectory penalty term to guide the drone to fly within the specified range. Therefore, the transmitter ST m The reward function of drone J is defined as in, Among them, z2 and z3 are the maximum flight distances of the drone on the x-axis and y-axis respectively; In DRL, the reward r[t] at a single time step constitutes the long-term return of the agent, so the final optimization problem is expressed as stC1~C8 Among them, the discount factor γ∈(0,1) allows decision makers to adjust the importance of current and future rewards; (4) State transition probability The resource allocation problem in problem P1 is modeled as an MDP. The state transition probabilities involved in the MDP Experience data is accumulated through continuous interaction between the intelligent agent and the environment, and calculations are continuously iterated based on the Bellman equation.
7. The multi-user energy harvesting resource allocation method based on UAV interference assistance according to claim 6 is characterized in that: Step S3 also includes: In the drone-assisted CR system, the optimization problem goal is to maximize the system's average confidentiality rate while meeting the primary user's QoS requirements. The combination of Double DQN and Dueling DQN, namely D3QN, is used as the training and target network, and the Prioritized Experience Replay PER mechanism is used, and the soft update method is set as the update rule. D3QN solves the problem of overestimation of Q value by using two network components to select actions and evaluate the value of the action, and separates two paths in the network architecture to calculate the state value function and the advantage function, respectively, and then combines them to estimate the Q value of each action, so that D3QN can more effectively learn more important actions in different states. Let θ k and are the parameters of the decision network and target network of the kth agent, respectively. The weight parameters of the two branches in the Dueling network are w V and w A The output state value function and advantage function are represented by V(s t ;θ,w V ) and A(s t ,a t ;θ,w A ) indicates that the Q values output by the decision network and the target network are in, |A| is the size of the action space; The PER mechanism based on TD error is used in the sampling process, and its error e is defined as Among them, κ∈(0,1) is the discount factor of the error; In the PER mechanism, data with higher priority P(i) will be drawn from the experience pool D k The higher the probability of being selected as a training sample; the sampling priority P(i) is defined as Among them, φ is a constant between 0 and 1, which determines the influence of priority, u i is the priority of the i-th experience; B extracted by the kth agent k The loss function L(θ k )for Decision network parameters θ k Minimize the loss function L(θ) by using the gradient descent algorithm k ) update, after a certain time step, the target network parameters Update according to the soft update method.
8. The method for allocating multi-user energy harvesting resources based on UAV interference assistance according to claim 7 is characterized in that: This method uses algorithm time complexity to evaluate efficiency.
9. A multi-user energy harvesting resource allocation system based on drone interference assistance, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.
10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.
Citation Information
Cited By
Compensation method and system for optimizing underwater UUV communication Doppler effect
CN119628758A
