Cognitive communication resource allocation method under interference and eavesdropping attack based on P-MADDPG

Abstract

The invention relates to a cognitive communication resource allocation method under interference and eavesdropping attack based on P-MADDPG, and belongs to the field of cognitive communication resource allocation. The method comprises the following steps: (1) proposing a multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping; and (2) providing a P-MADDPG resource allocation method based on the combination of a PER (Priority Empirical Replay) mechanism and a multi-agent reinforcement learning algorithm MADDPG of a CTDE architecture so as to cope with interference attacks and eavesdropping attacks of the model in the step (1) in a transmission stage and realize cognitive communication resource allocation. According to the method, the problem of distributing the transmitting power, the transmission time and the auxiliary interference power for the multi-user distributed energy acquisition cognitive communication network under interference and eavesdropping attacks can be solved, and the average secrecy rate of the SU network is improved.

Description

Technical Field

[0001] The invention belongs to the field of cognitive communication resource allocation, and in particular relates to a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG. Background Art

[0002] In actual networks, the network will not only be subject to interference attacks, but also other attacks, such as eavesdropping attacks. Eavesdropping attacks illegally intercept and decode the confidential information of ST, which makes the transmission of ST confidential information unprotected. Moreover, interference attacks and eavesdropping attacks may exist in the network at the same time, and both will greatly reduce the information transmission quality and security of SU network. Therefore, how to improve the performance of SU network under the coexistence of interference and eavesdropping attacks through reasonable resource allocation is the focus of current research.

[0003] In the existing literature, the PLS of the energy-harvesting NOMA cognitive communication network is studied. By adjusting the time allocation between the energy harvesting phase and the information transmission phase, the effective confidentiality throughput under two eavesdropping scenarios is maximized, but the existing literature only considers the single-user scenario. The existing literature studies the IRS-assisted cooperative interference communication model. Under the constraints of the minimum security rate and inaccurate channel state information, a DDPG-based joint optimization of the base station transmit beam and the IRS phase shift matrix is ​​proposed to maximize the system energy efficiency. However, this literature only models the environment as a single-agent system, which requires unified control by the base station, aggravates the system overhead, and does not consider the collusion attack of interference and eavesdropping. The existing literature also proposes a backscattering enhancement strategy based on IRS, which maximizes the system confidentiality rate by jointly designing the reflection coefficient of IRS and the beamforming of the base station to ensure the transmission security of MIMO under interference and eavesdropping, but this literature only considers the scenario of a single eavesdropper. Therefore, how to focus on the transmission power, transmission time and auxiliary interference power allocation of multi-user distributed energy-harvesting cognitive communication networks under interference and eavesdropping attacks, and improve the average confidentiality rate of SUs networks, is the focus of existing research. Summary of the invention

[0004] The purpose of the present invention is to provide a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG for the transmission power, transmission time and auxiliary interference power allocation problems of a multi-user distributed energy harvesting cognitive communication network under interference and eavesdropping attacks, so as to improve the average confidentiality rate of the SU network.

[0005] To achieve the above object, the technical solution of the present invention is: a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG, comprising:

[0006] (1) A multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is proposed;

[0007] (2) A P-MADDPG resource allocation method (P-MADDPG: Prioritized Experience Replay (PER) combined with Multi-Agent Deep Deterministic Policy Gradient (MADDPG)) based on the Prioritized Experience Replay (PER) mechanism and the CTDE (centralized training and decentralized execution) architecture is proposed to deal with the interference attack and eavesdropping attack of the model in step (1) during the transmission phase and realize cognitive communication resource allocation.

[0008] In one embodiment of the present invention, in step (1), a multi-user distributed energy harvesting cognitive communication network model in which interference and eavesdropping coexist is considered, and the existence of multiple eavesdroppers and probability-based interference attacks are considered; considering the collusion attack of interference and eavesdropping, the eavesdropping channel is degraded by introducing auxiliary interference nodes, and the DRL technology is used as a means. The goal is to jointly optimize the transmission power, transmission time and auxiliary interference power, and at the same time maximize the optimization problem of the average confidentiality rate of the SUs network under the constraints of the user service quality QoS requirements of the primary user PU network and the secondary user SU network.

[0009] In one embodiment of the present invention, in step (1), the system model of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is constructed as follows:

[0010] A multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is proposed, which includes three parts: PU network, SU network and malicious attack network. Among them, the PU network contains 1 primary user transmitter PT and 1 primary user receiver PR. The SU network contains N secondary user transmitters ST, 1 secondary user base station SBS and 1 friendly jammer FJ. ST dynamically accesses the PU network spectrum in the underlying or interleaved spectrum access mode and transmits data to the SBS in time division multiple access TDMA mode. FJ degrades the eavesdropping channel by transmitting auxiliary interference noise. It is assumed that FJ generates noise using a pseudo-random sequence known to the legitimate user, which enables the legitimate user to eliminate this noise, while Eve knows the pseudo-random sequence, so Eve cannot eliminate the noise. The malicious attack network contains B malicious eavesdroppers Eve and 1 malicious jammer MJ. Eve eavesdrops on the confidential information of ST received by SBS, and MJ interferes with SBS through attack strategies. Both are committed to reducing the transmission quality and confidentiality performance of the SU network.

[0011] In one embodiment of the present invention, in step (1), the channel gain model of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is constructed as follows:

[0012] Considering that SBS, ST, FJ, Eve, and MJ are all distributed in the coverage area of ​​the main network, define ST n For the nth ST, ST v For the vth ST, Eve b is the b-th Eve, where n∈{1,2,...,N}, v∈{1,2,...,N} and v≠n, b∈{1,2,...,B}, (x PT ,y PT )、(x PR ,y PR )、(x SBS ,y SBS ) represent the position coordinates of PT, PR and SBS respectively, Respectively represent ST n 、ST v 、Eve b The position coordinates, (x FJ ,y FJ )、(x MJ ,y MJ ) represent the position coordinates of FJ and MJ respectively. Considering that SBS, ST, FJ, Eve, and MJ are all distributed in the coverage area of ​​the main network, define ST n For the nth ST, ST v For the vth ST, Eve b is the b-th Eve, where n∈{1,2,...,N}, v∈{1,2,...,N} and v≠n, b∈{1,2,...,B}, (x PT ,y PT )、(x PR ,y PR )、(x SBS ,y SBS ) represent the position coordinates of PT, PR and SBS respectively, Respectively represent ST n , ST v 、Eve b The position coordinates, (x FJ ,y FJ )、(x MJ ,y MJ ) represent the position coordinates of FJ and MJ respectively; the channel model is expressed as follows:

[0013]

[0014] Among them, λ is the Rician factor, which is used to describe the signal fading in the wireless channel and represents the difference in signal power between the LOS path and the scattering path; θ:U(0,1) represents the phase of the signal arriving on the LOS path, which takes a value between 0 and 1 and obeys a uniform distribution; CN(·) is a statistical representation, representing a ring-symmetric complex Gaussian random variable; σ 2 =10 -ζ , σ 2 Depends on the path loss, ζ represents a comprehensive path loss indicator;

[0015] The channel gain model g at time slot t p,n , g n,r , g v,n 、h n,s 、h f,n 、e n,b 、e f,b 、e p,b , g m,s , g m,n Represents PT→ST n , ST n →PR, ST v →ST n , ST n →SBS、FJ→ST n , ST n →Eve b 、FJ→Eve b ,PT→Eve b 、MJ→SBS、MJ→ST n The channel power gain of the eavesdropping channel can be obtained by performing statistical estimation on the information of the eavesdropper channel. Therefore, the channel gain set G(t) perceived by the ST of the entire SU network in time slot t is obtained as follows:

[0016]

[0017] Among them, G n (t) is the set of channel gains perceived by STn.

[0018] In one embodiment of the present invention, in step (1), the attack model of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is constructed as follows:

[0019] Interference model: For the single-channel model, consider the existence of an energy-limited probabilistic interference-based MJ. MJ is the same as ST and has spectrum sensing capability. It considers perfect sensing, but cannot detect the state of ST. It senses the spectrum state before launching an attack. If the spectrum is idle, it launches an interference attack on the SU network with probability β, β∈[0,1]. Use P J (t) represents the interference power in time slot t, Indicates the maximum interference power that MJ can take; if the channel is busy, MJ will remain silent;

[0020] Eavesdropping model: Each Eve has a radio receiver and always tries to eavesdrop on ST n The data transmitted to the SBS is not eavesdropped by collusion between eavesdroppers, that is, each Eve decodes the confidential message individually, and then considers the Eve with the best receiving performance; the collusion attack between interference and eavesdropping is considered, that is, the eavesdropper Eve also uses the signal processing method to eliminate the interference signal of MJ. In addition, MJ also promotes eavesdropping by reducing the signal-to-noise ratio SINR at the SBS.

[0021] In one embodiment of the present invention, in step (1), the time slot and energy model of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is constructed as follows:

[0022] 1) Time slot model

[0023] ST sends data to SBS in TDMA mode, that is, ST sends data in sequence, and the length of each time slot is the same and fixed to T. The time slot T is divided into two stages:

[0024] Perception stage, ST n PT is sensed at the beginning of the time slot. Considering the case of perfect perception, the ST spectrum sensing set S(t) at time slot t is [I1(t), I2(t), ..., I N (t)] T , where I n (t) is expressed as ST n The perception result, I n (t)∈{0(idle),1(busy)}; the channel occupancy state transition also follows a two-dimensional Markov chain; after sensing, ST n A transmission decision will be made; assuming that the time consumed in the entire stage is τ and the energy consumed is E τ , are all fixed constants;

[0025] In the transmission phase, after the sensing is completed, ST n Adopting the transmission strategy, the auxiliary jammer FJ sends interference noise to the eavesdropper Eve; considering the transmission mode of mixed bottom layer transmission and interleaved transmission, specifically, when PT occupies the channel, that is, In (t) = 1, ST n The PU channel is accessed in the bottom layer mode for data transmission. When the PT channel is idle, that is, I n (t) = 0, ST n Access the PU channel in interleaved mode for data transmission; during the transmission phase, ST n , FJ according to the system channel status I n (t), channel gain G n (t) and ST n Battery Level B n (t) Determine the transmission time α n (t), transmission power P n (t) and auxiliary interference power P n F (t); at this time, the transmission time strategy set α(t), the transmission power strategy set P(t) and the auxiliary interference power strategy set P of STs and FJ are F (t) is expressed as:

[0026]

[0027] Among them, α n (t)∈[0,T-τ], and P n (t)∈[0,P max ], P max , ST n , the maximum transmission power of FJ;

[0028] 2) Energy Model

[0029] ST has the same structure and function, and has an energy harvesting circuit; except for ST, all devices are powered by the grid; ST is powered by a rechargeable battery with a capacity of B max ; Limited by the hardware structure, it follows the mode of acquisition-storage-transmission, that is, the current acquisition needs to be stored in the battery and used in the subsequent transmission stage; define a two-bit variable H n (t) represents the working mode of STn in time slot t:

[0030]

[0031] On this basis, further analysis of ST n The RF energy collected at time slot t

[0032]

[0033] When ST vWorking in transmission mode, TDMA transmission mode allows ST in the waiting transmission stage to n Energy harvesting; therefore, the above formula also includes ST n Energy collected during the waiting period; the first term in the above formula is ST n The collected ST v and the energy of FJ, the second term is ST n The energy collected in PT or MJ, P T is the PT transmission power, which is set as a fixed constant; define a binary variable J(t)∈{0(interference), 1(sleep)}, indicating whether MJ initiates interference in the current time slot; at this time, the ST energy collection set in the SU network is E(t)=[E1(t),E2(t),...,E N (t)] T , use B n (t) Define the time slot ST n The battery level, B n (t)∈[0,B max ]; Assuming that the battery is ideal, only the sensing and transmission phases consume energy. When the energy harvesting exceeds the maximum battery capacity B max When , the excess energy will overflow, and the battery level at time slot t+1 is:

[0034] B n (t+1)=min{B max ,B n (t)+E n (t)-P n (t)α n (t)-Z n (t)E τ}

[0035]

[0036] Among them, Z n (t) is the spectrum sensing indicator, so the battery state set B(t) of ST at time slot t is:

[0037] B(t)=[B1(t),...,B2(t),...,B N (t)] T .

[0038] In one embodiment of the present invention, in step (1), the optimization problem of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is specifically as follows:

[0039] Considering the operation process of the multi-user distributed energy harvesting cognitive communication network model with coexistence of interference and eavesdropping, the model is analyzed according to the two modes of bottom-layer transmission and interleaved transmission. When PT exists at time slot t, that is, bottom-layer mode transmission, the signal received at the SBS is:

[0040]

[0041] The first item is the ST received by SBS. n The transmitted signal, x n (t) represents the transmission signal from ST n ; The second term is the interference signal of PT received by SBS, x p (t) represents the transmission signal to PT; is a product with zero mean and Then SBS receives ST at time slot t n The SINR of a signal is described as:

[0042]

[0043] Then ST at time slot t n The achievable rate R at SBS n,sbs (t) is described as:

[0044]

[0045] When PT is present, ST n It may interfere with the transmission of PT. In order not to affect the normal transmission of PT, the PU network needs to be analyzed at the same time. The signal received by PR at time slot t is:

[0046]

[0047] is a product with zero mean and The SINR received at the PR is:

[0048]

[0049] Similarly, now for the eavesdropper Eve b Conduct analysis, Eve b The overheard signal is described as:

[0050]

[0051] Among them, the first item is the ST n signal, the second is the PT signal, and the third is the FJ emission interference noise. is a product with zero mean and Variance complex Gaussian noise; then Eve b ST eavesdropped at time slot t n The signal-to-noise ratio SINR is described as:

[0052]

[0053] Then ST at time slot t n In Eve b The achievable rate R n,b (t) is described as:

[0054]

[0055] According to the definition of confidentiality rate, at this time, SBS in time slot t is about ST n The confidentiality rate is derived as:

[0056]

[0057] Where [x] + represents max{x,0}; at the same time, the SINR set at time slot t is obtained as:

[0058]

[0059] Similarly, when PT does not exist, that is, ST n When using interleaved mode transmission, SBS and Eve b SINR received at:

[0060]

[0061] Define a binary variable J(t)∈{0,1} to indicate whether the SBS is attacked by MJ in time slot t;

[0062] At this time, the time slot SBS is about ST n The confidentiality rate is derived as:

[0063]

[0064] The goal of the optimization problem of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is to maximize the average confidentiality rate of the SU network system under interference and eavesdropping attacks by allocating power and time resources. The final optimization problem P1 is expressed as:

[0065]

[0066] Among them, L is the number of time slots, C1 is ST n Transmission time constraint; C2 is ST nTransmission power constraint; C3 is FJ interference power constraint; C4 is ST n Battery energy constraint; C5 is the minimum SINR received by PR and SBS p,r,min 、SINR n,sbs,min Constraint; C6 is ST n The confidentiality rate to SBS must be greater than the minimum confidentiality rate R sec,min Require.

[0067] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0068] Compared with the prior art, the present invention has the following beneficial effects: The present invention considers the existence of interference and eavesdropping collusion attacks in energy harvesting cognitive communication networks. In the scenario of collusion attack, limited user energy and user service quality QoS constraint, STs adopt an interleaved-bottom mixed spectrum access mode and access the channel in turn in TMDA mode. By introducing auxiliary interference nodes, degrading the eavesdropping channel, and by reasonably allocating the transmission power, transmission time and auxiliary interference node power, the system average confidentiality rate is maximized. First, according to the network model and the definition of confidentiality rate, the optimization problem is obtained. In view of the optimization problem of resource coupling, unknown attack mode, dynamically changing channel information, and the traditional method solving complex situations, a P-MADDPG continuous action control resource allocation method based on CTDE architecture is proposed. Secondly, in this method, ST and auxiliary interference FJ are jointly modeled as intelligent agents, and the optimal resource allocation strategy is found by designing a reasonable state space, a continuous action space and a reward function to improve the ST network performance. Finally, simulation is carried out. The results show that the average confidentiality rate of the proposed system based on P-MADDPG method is improved by 33.31% and 43.69% compared with I-DDPG and DDPG respectively. At the same time, the results also show that the average confidentiality rate of the proposed method is better than the comparison method under different parameters. In summary, the above results prove the effectiveness of the proposed resource allocation method in the collusion attack scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 The present invention considers a cognitive communication network model for energy harvesting under interference and eavesdropping attacks.

[0070] Figure 2 A time-slot model for energy harvesting cognitive communication networks under jamming and eavesdropping attacks.

[0071] Figure 3 This is the structure diagram of the resource allocation algorithm based on P-MADDPG.

[0072] Figure 4 is the learning rate and the average confidentiality rate.

[0073] Figure 5 is the number of training rounds and the average confidentiality rate under collusion attack.

[0074] Figure 6 is the number of training rounds and the average confidentiality rate under non-collusion attack.

[0075] Figure 7 is the maximum malicious interference power and the average confidentiality rate.

[0076] Figure 8 is the maximum auxiliary interference power and the average confidentiality rate.

[0077] Fig. 9 is the maximum transmission power and average confidentiality rate of ST.

[0078] Fig.10 is the number of eavesdroppers and the average confidentiality rate.

[0079] Fig.11 is the number of STs and the average confidentiality rate.

[0080] Fig.12 It is ST's maximum battery capacity and average confidentiality rate. DETAILED DESCRIPTION

[0081] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0082] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0083] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0084] The present invention provides a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG, comprising:

[0085] (1) A multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is proposed;

[0086] (2) A P-MADDPG resource allocation method based on the Prioritized Experience Replay (PER) mechanism combined with the multi-agent reinforcement learning algorithm MADDPG based on the CTDE architecture is proposed to deal with the interference attack and eavesdropping attack of the model in step (1) during the transmission phase and realize cognitive communication resource allocation.

[0087] The following is the specific implementation process of the present invention.

[0088] 1. System model

[0089] Considering the energy harvesting cognitive communication network model under interference and eavesdropping, Figure 1 The model consists of three parts: PU network, SU network, and malicious attack network. Among them, the PU network contains 1 PT and 1 PR. The SU network contains N STs, 1 secondary base station (SBS), and 1 friendly jammer (FJ). ST dynamically accesses the PU network spectrum in the underlying or interleaved spectrum access mode and transmits data to the SBS in time division multiple access (TDMA). FJ degrades the eavesdrop channel by transmitting auxiliary interference noise. Assume that FJ generates noise using a pseudo-random sequence known to legitimate users, which allows legitimate users to eliminate this noise, but Eves does not know the sequence, so Eve cannot eliminate the noise. The malicious attack network contains B malicious eavesdroppers (Eve) and 1 malicious jammer (MJ). Eve eavesdrops the confidential information of ST received by SBS. The jammer MJ interferes with SBS through certain attack strategies. Both are committed to reducing the transmission quality and confidentiality performance of the SU network.

[0090] 1.1 Channel Gain Model

[0091] Considering that SBS, ST, FJ, Eve, and MJ are all distributed in the coverage area of ​​the main network, define ST n For the nth ST, ST v For the vth ST, Eve b is the b-th Eve, where n∈{1,2,...,N}, v∈{1,2,...,N} and v≠n, b∈{1,2,...,B}, (x PT ,y PT )、(x PR ,y PR )、(x SBS ,y SBS ) represent the position coordinates of PT, PR and SBS respectively, Respectively represent STn , ST v 、Eve b The position coordinates, (x FJ ,y FJ )、(x MJ ,y MJ ) represent the position coordinates of FJ and MJ respectively; the channel model is expressed as follows:

[0092]

[0093] Among them, λ is the Rician factor, which is used to describe the signal fading in the wireless channel and represents the difference in signal power between the LOS path and the scattering path; θ:U(0,1) represents the phase of the signal arriving on the LOS path, which takes a value between 0 and 1 and obeys a uniform distribution; CN(·) is a statistical representation, representing a ring-symmetric complex Gaussian random variable; σ 2 =10 -ζ , σ 2 Depends on the path loss, ζ represents a comprehensive path loss indicator;

[0094] Similarly, the channel gain model g at time slot t can be obtained p,n , g n,r , g v,n 、h n,s 、h f,n 、e n,b 、e f,b 、e p,b , g m,s , g m,n , representing PT→ST n , ST n →PR, ST v →ST n , ST n →SBS、FJ→ST n , ST n →Eve b 、FJ→Eve b ,PT→Eve b 、MJ→SBS、MJ→ST n The channel power gain of the eavesdropping channel can be obtained by performing statistical estimation on the information of the eavesdropper channel. Therefore, the channel gain set G(t) perceived by the ST of the entire SU network in time slot t can be obtained as follows:

[0095]

[0096] Among them, Gn (t) is ST n The set of perceived channel gains.

[0097] 1.2 Attack Model

[0098] Interference model: For the single-channel model, consider the existence of an energy-limited probabilistic interference-based MJ. MJ is the same as ST and has spectrum sensing capabilities. It considers perfect sensing, but cannot detect the state of ST. Before launching an attack, it senses the spectrum state. If the spectrum is idle, it launches an interference attack on the SU network with probability β, β∈[0,1]. Use P J (t) represents the interference power in time slot t, Indicates the maximum interference power that MJ can take. If the channel is busy, MJ will remain silent.

[0099] Eavesdropping model: Each Eve has a radio receiver and always tries to eavesdrop on ST n The data transmitted to the SBS is eavesdropped by the eavesdroppers without collusion, that is, each Eve decodes the confidential message individually, and then considers the Eve with the best receiving performance. It is worth mentioning that the present invention considers the collusion attack initiated by interference and eavesdropping, that is, the eavesdropper Eves also use signal processing and other methods to eliminate the interference signal of MJ. In addition, MJ also promotes eavesdropping by reducing the SINR at the SBS.

[0100] 1.3 Time Slot and Energy Model

[0101] 1) Network time slot model

[0102] ST sends data to SBS in TDMA mode, that is, ST sends data in sequence. Each time slot has the same length and is fixed to T. Furthermore, time slot T can be divided into two stages, as follows: Figure 2 shown.

[0103] Perception stage, ST n PT is sensed at the beginning of the time slot, considering the case of perfect perception. The ST spectrum sensing set S(t) at time slot t = [I1(t), I2(t), ..., I N (t)] T Among them, I n (t) is expressed as ST n The perception result, I n (t)∈{0(idle),1(busy)}. The channel occupancy state transition also follows a two-dimensional Markov chain. After sensing, ST n A transmission decision will be made. Assume that the time consumed in the whole stage is τ and the energy consumed is E τ , are all fixed constants.

[0104] In the transmission phase, after the sensing is completed, ST n The transmission strategy is adopted, and the auxiliary jammer FJ sends interference noise to the eavesdropper Eves. The present invention considers the transmission mode of mixed bottom layer transmission and interleaved transmission. Specifically, when PT occupies the channel, that is, I n (t) = 1, ST n The PUs channel is accessed in the bottom layer mode for data transmission. When the PT channel is idle, that is, I n (t) = 0, ST n Access the PUs channel in an interleaved mode for data transmission. During the transmission phase, ST n , FJ according to the system channel status I n (t), channel gain G n (t) and ST n Battery Level B n (t) Determine the transmission time α n (t), transmission power P n (t) and auxiliary interference power P n F (t). At this time, the transmission time strategy set α(t), the transmission power strategy set P(t) and the auxiliary interference power strategy set P of ST and FJ are F (t) can be expressed as:

[0105]

[0106] Among them, α n (t)∈[0,T-τ], and P n (t)∈[0,P max ]、 P max , For ST n , the maximum transmission power of FJ.

[0107] (2) Energy model

[0108] The ST has the same structure and function, and has an energy harvesting circuit. All devices except the ST are powered by the grid. The ST is powered by a rechargeable battery with a capacity of B max Limited by the hardware structure, the acquisition-storage-transmission mode is followed, that is, the current acquisition needs to be stored in the battery and used in the subsequent transmission stage. Define a binary variable H n (t) indicates ST n Working mode in time slot t:

[0109]

[0110] On this basis, we can further analyze the STn The RF energy collected at time slot t

[0111]

[0112] It is worth noting that when ST v Working in transmission mode, TDMA transmission mode allows ST in the waiting transmission stage to n Therefore, the above formula also includes ST n Energy collected during the waiting period. The first item is ST n The collected ST v and the energy of FJ. The second term is ST n The energy collected in PT or MJ, P T is the PT transmission power, which is set as a fixed constant. Define a binary variable J(t)∈{0 (interference), 1 (sleep)}, indicating whether MJ launches interference in the current time slot. At this time, the ST energy collection set in the SU network is E(t)=[E1(t),E2(t),...,E N (t)] T . Use B n (t) Define the time slot ST n The battery level, B n (t)∈[0,B max ]. Assuming the battery is ideal, only the sensing and transmission phases consume energy. When the energy harvesting exceeds the maximum battery capacity B max When , the excess energy will overflow. Then the battery level at time slot t+1 is derived as:

[0113] B n (t+1)=min{B max ,B n (t)+E n (t)-P n (t)α n (t)-Z n (t)E τ}

[0114]

[0115] Among them, Z n (t) is the spectrum sensing indicator. Therefore, the battery state set B(t) of ST at time slot t is:

[0116] B(t)=[B1(t),...,B2(t),...,B N (t)] T

[0117] 2. Analysis of resource allocation problem for maximizing average confidentiality rate

[0118] This section analyzes the operation process of the network model and makes a specific deduction for the optimization problem, analyzing it according to the two modes of bottom-layer transmission and interleaved transmission. When PT exists at time slot t, that is, bottom-layer mode transmission, the signal received at the SBS is:

[0119]

[0120] The first item is the ST received by SBS. n The transmitted signal, x n (t) represents the transmission signal from ST n ; The second term is the interference signal of PT received by SBS, x p (t) represents the transmission signal to PT; is a product with zero mean and Then the SBS receives ST at time slot t. n The SINR of a signal can be described as:

[0121]

[0122] Then ST at time slot t n The achievable rate R at SBS n,sbs (t) can be described as:

[0123]

[0124] When PT is present, ST n It may interfere with the transmission of PT. In order not to affect the normal transmission of PT, the PUs network needs to be analyzed at the same time. The signal received by PR at time slot t is:

[0125]

[0126] is a product with zero mean and The SINR received at the PR is:

[0127]

[0128] Similarly, now for the eavesdropper Eve b Conduct analysis, Eve b The eavesdropped signal can be described as:

[0129]

[0130] Among them, the first item is the ST n signal, the second is the PT signal, and the third is the FJ emission interference noise. is a product with zero mean and The variance of complex Gaussian noise. Then Eve b ST eavesdropped at time slot t n The signal-to-noise ratio SINR can be described as:

[0131]

[0132] Then ST at time slot t n In Eve b The achievable rate R n,b (t) can be described as:

[0133]

[0134] According to the definition of confidentiality rate, at this time, SBS in time slot t is about ST n The confidentiality rate is derived as:

[0135]

[0136] Where [x] + represents max{x,0}. At the same time, the SINR set at time slot t is obtained as:

[0137]

[0138] Similarly, when PT does not exist, that is, ST n When using interleaved mode transmission, SBS and Eve b SINR received at:

[0139]

[0140] Define a binary variable J(t)∈{0,1} to indicate whether the SBS is attacked by MJ in time slot t.

[0141] At this time, the time slot SBS is about ST n The confidentiality rate is derived as:

[0142]

[0143] The goal of the present invention is to maximize the average confidentiality rate of the SU network system under interference and eavesdropping attacks by reasonably allocating power and time resources. The optimization problem P1 finally established can be expressed as:

[0144]

[0145] Where L is the number of time slots. C1 is ST n Transmission time constraint; C2 is ST n Transmission power constraint; C3 is FJ interference power constraint; C4 is STn Battery energy constraint; C5 is the minimum SINR received by PR and SBS p,r,min 、SINR n,sbs,min Constraint; C6 is ST n The confidentiality rate to SBS must be greater than the minimum confidentiality rate R sec,min The present invention aims to find the optimal transmission time, transmission power and auxiliary interference power allocation scheme under the conditions of C1 to C6, improve the average confidentiality rate of SU network, and thus realize SU network secure transmission.

[0146] 3. MADRL-based resource allocation framework under jamming and eavesdropping attacks

[0147] 3.1 Mapping from Energy Harvesting Cognitive Communication Network to MADRL

[0148] For the above optimization problem, various resources in the environment are coupled with each other, and unknown interference and eavesdropping attack behaviors and channel information changes lead to dynamics and uncertainty, which increases the difficulty of handling the problem. Model-free deep reinforcement learning is a dynamic programming tool that can handle decision-making problems by obtaining the optimal solution in large-scale dynamic scenarios through a dynamic trial and error process without knowing the attack model.

[0149] In the MADRL resource allocation framework, we first need to model the multi-agent and environment and design appropriate states, action spaces, and effective reward functions. This multi-user resource allocation task is modeled as POMDP, using Six-tuple representation. At time slot t, agent n is based on the current state environment Get local observation And according to the strategy π(·|s t ) Execute an action Then get rewarded Current Status Based on the state transition probability function Transfer to the next state. In order to adapt to the resource allocation problem of the present invention, a detailed explanation of the main elements of the multi-agent system in this model is now given:

[0150] Agent: In the energy harvesting cognitive communication network, ST n Together with FJ, they are modeled as one intelligent agent, with a total of N intelligent agents.

[0151] State Space The agent needs to know the channel information, including PT state, ST nThe initial battery energy and channel gain of the agent are as much as possible. With sufficient channel information, the agent can make appropriate action decisions based on the current state. Therefore, in a network where interference and eavesdropping coexist, the state space includes the spectrum sensing set of ST, the channel gain set, the SINR set, and the battery level state of ST. At time slot t, the state space Designed for:

[0152]

[0153] Local Observation Space Represents the agent ST n In the local observation space of time slot t, ST n The spectrum sensing set, channel gain set, SINR set and ST n The battery level state composition, local observation space Can be designed as:

[0154]

[0155] Action Space In the resource allocation scenario, agent n reasonably allocates transmission time, transmission power, and auxiliary interference to achieve the optimal average confidentiality rate performance under interference and eavesdropping attacks. Action space of agent n The design is as follows:

[0156]

[0157] Reward Space t n :Agent n based on local observations Make a move After that, they interact with the environment and get rewards. The goal of this invention is to maximize the average confidentiality rate of the entire SU network under the relevant constraints, so the reward function must include R n,sec (t), the larger the reward function, the greater the R n,sec The larger (t) is, the better the action decision made by the agent is. Considering the optimization problem and introducing the penalty term, the reward function r of agent n is t n The design is as follows:

[0158]

[0159] in,

[0160]

[0161] ρ and χ represent penalty values. Introducing penalty terms can stimulate the agent to better learn unknown environments.

[0162] 3.2. Design of resource allocation algorithm framework based on P-MADDPG

[0163] For the power and time allocation problem, it is more appropriate to regard it as a continuous action problem, so that the agent can better adapt to different environments and task requirements. MADDPG is based on the Actor-Critic framework and deterministic strategy, and is also an advanced algorithm that can solve continuous action problems. It can handle POMDP problems with high-dimensional actions and state spaces, and is applied to complex multi-agent scenarios that traditional DRL algorithms cannot handle. In addition, in multi-agent reinforcement learning, the agent makes action decisions only based on local observations. From the perspective of any agent, the network environment is non-stationary, which violates the learning stability and convergence required by the Markov assumption. Therefore, the present invention adopts the MADDPG algorithm of the CTDE framework. Specifically, during the training process, a Critic network that can obtain the global situation is introduced to guide the training of the Actor. When making a decision, the agent only needs to make the correct action based on local information.

[0164] During the training phase, MADDPG uses the experience pool The experience in the pool D is uniformly randomly sampled. Random sampling does not take into account the importance of the experience. There is a risk that a small amount of high-value information will not be sampled for a long time. At the same time, the capacity of the experience pool D is fixed. When the experience is full and new experience needs to be stored, the old experience must be deleted. The deleted experience may contain key information. The present invention applies PER technology to DQN to accelerate network convergence. PER technology gives each experience a priority, which reflects the importance of experience for algorithm learning. When the experience is played back, targeted sampling is performed according to the importance of the experience. Therefore, the present invention improves the random sampling in MADDPG and proposes a P-MADDPG resource allocation method based on PER and MADDPG based on the CTDE architecture.

[0165] The proposed P-MADDPG algorithm structure diagram is Figure 3 In order to simplify the representation, the agent set is represented as Agent n is composed of 4 neural networks, including target policy network, online policy network, target Q network and online Q network. The target policy network and online policy network belong to the Actor network, while the target Q network and online Q network belong to the Critic network. In addition, there is an experience replay pool. Used to store the experience transformation tuples of the training phase. Define μ = {μ1,μ2,...,μ N} is the set of deterministic strategies of all agents in the online strategy network, μ′={μ1′,μ2′,...,μ′N} is the set of deterministic strategies in the target strategy network, and θ={θ1,θ2,...,θ N} represents the parameters of the deterministic strategy μ, θ′={θ1′,θ2′,...,θ′ N} are the parameters of the deterministic strategy μ′. The online strategy network of agent n at time slot t is determined by the deterministic strategy μ n The local observation Mapping to specific actions Right now In order to explore the optimal action strategy, random noise is added to the action strategy, so the actual action is described as:

[0166]

[0167] Agent n follows the action strategy Interact with the environment to gain Next observation The agent stores its own experience tuples in the replay pool middle, The training experience tuple (s t ,a t ,r t ,s t+1 ) contains the current state, action, reward and next state of all agents. The Critic network of P-MADDPG is t n The action value is evaluated in a similar way to the value evaluation method of the Critic network in the DQN network. The centralized action-value function Q(s) is calculated by inputting the state and action. t ,a t ), it is worth noting that the states and actions here refer to the states and actions of all agents, that is, (s t ,a t ). The centralized action-value Q function can be calculated by the Bellman equation:

[0168]

[0169] Where γ is the discount factor. Q ω (s,a) is used to evaluate the current state s t Take action a t The effect is to help the online policy network of agent n improve its policy.

[0170] PER sets a priority for each experience tuple based on TD-error. The experience tuple with a larger TD-error has a greater probability of being sampled. Use δn represents its TD-error, δ n Calculated as:

[0171]

[0172] Among them, ω n ,ω n ′ represents the online critic network and target critic network parameters of agent n. Experience tuple x n The priority can be calculated by the following formula:

[0173] p n =|δ n |+ζ

[0174] where ζ is a small positive constant, ensuring that δ n It can also be sampled normally when it is 0. During the training phase, the network parameters of Actor and Critic are obtained by replaying the experience pool. The sampling batch size using PER technology is The experience tuple is used to update, and this process will continue until the end. n The probability of being selected P(x n )With the priority monotony, P(x n ) is described as:

[0175]

[0176] Among them, α1∈[0,1] is used to adjust the importance of the priority in the sampling process. α1=0 corresponds to the original uniform random sampling, and α1=1 corresponds to the direct use of the TD-error value. In order to alleviate the problem of PER causing changes in the sample sampling frequency and causing oscillations in the training phase, the importance sampling weight is introduced:

[0177]

[0178] represents the capacity of the experience replay pool, and α2 is the degree of error correction. Then, the Critic network will learn the action value Q function corresponding to the optimal strategy by minimizing the loss. The Critic network loss is described as:

[0179]

[0180] y is the sum of the immediate reward and the discounted target Q value obtained by the target Q network:

[0181]

[0182] in, For ST nThe target policy network outputs actions. The Actor network of each agent updates the parameters θ and adjusts the strategy based on the centralized action-value Q function and observation information calculated by the Critic network to optimize the objective function J(θ n ), the optimal strategy of agent n can be expressed as:

[0183]

[0184] Parameter θ n Directly in the Actor network Deterministic policy gradient direction adjustment:

[0185]

[0186] It consists of two parts: the gradient of the action value Q function from the Critic network and its own deterministic policy action gradient To improve the stability of learning, as the online policy network and the online Q network are updated in real time, the target policy network parameters θ n ′ and the parameter ω of the target Q network n ′ will be updated by the soft update rule:

[0187]

[0188] Among them, ξ a ∈[0,1],ξ c ∈[0,1] is the soft update factor of the Actor and Critic networks. After training, agent n can choose its own actions based on local observations without knowing the information of other agents. The resource allocation algorithm flow under interference attack based on P-MADDPG is shown in Table 1.

[0189] Table 1 Resource allocation algorithm based on P-MADDPG interference and eavesdropping attacks

[0190]

[0191]

[0192] 3.3 Algorithm complexity and convergence analysis

[0193] Analyzing the complexity of an algorithm is crucial to the practicality of the algorithm. The present invention analyzes the complexity of the proposed resource allocation method based on P-MADDPG from the following three aspects.

[0194] (1) Number of trainable parameters

[0195] In P-MADDPG, the input of the centralized Q function is the observations and actions of all agents, and the observation and action spaces of all agents in this model are the same, defined as σ and ω respectively. Then the number of trainable parameters of P-MADDPG is Where N represents the number of agents.

[0196] (2) Total number of neural networks

[0197] In P-MADDPG, each agent contains 2 Actor networks and 2 Critic networks, so the total number of neural networks is 2×(N(l Actor +l Critic )), l Actor , l Critic Represents the Actor and Critic network for each agent.

[0198] (3) Computational complexity

[0199] In P-MADDPG, the determination of the agent's actions is related to the Actor network and the Critic network. The number of network layers in the Actor network is defined as L Actor , the number of neurons in the i-th layer is The computational complexity of the Actor network can be expressed as:

[0200]

[0201] Similarly, the number of network layers and the number of neurons in the Critic network are defined as L Critic and Then the computational complexity of the Critic network can be expressed as:

[0202]

[0203] Therefore, the total computational complexity is:

[0204]

[0205] Among them, B N is the mini-batch sampling size, E p is the number of training rounds, and L is the maximum number of training steps per round.

[0206] Different from the algorithm of direct convergence analysis, for the policy-based DRL algorithm, especially when it is extended to a multi-agent system, multiple agents interact with the environment during the training process, and the hyperparameters also have a great influence on the convergence of the learning algorithm. It is difficult to prove the convergence of the algorithm through theoretical analysis. Therefore, the present invention evaluates the convergence of the proposed algorithm through simulation.

[0207] 4. Experimental simulation and result analysis

[0208] 4.1 Simulation settings

[0209] In this section, the effectiveness of the proposed P-MADDPG method is verified by giving simulation results. All simulations in this section are performed under the framework of Python3.9 and Pytorch1.7.1-GPU. In the energy harvesting cognitive communication network considered, there are 1 PT, 1 PR, 3 STs, 1 SBS, 1 FJ, 3 Eves and 1 MJ. The change of the PT occupied channel state follows an independent two-dimensional Markov chain. 00 and p 11 Randomly selected from [0,0.3] and [0,0.4]. The positions of PT, PR, STs, SR, FJ, Eves and MJ are randomly distributed in the area of ​​150m*150m, and the distance between STs and SBS is randomly distributed between 20m-40m. The main simulation parameters are given in Table 2:

[0210] Table 2 Energy harvesting cognitive communication network simulation parameters

[0211]

[0212] In order to evaluate the effectiveness of the proposed method, the present invention compares the proposed method with the following methods:

[0213] Independent-DDPG (I-DDPG) method: In the I-DDPG method, the input of the Critic network only contains its own state and action, and each Agent executes the DDPG algorithm independently to maximize its own reward.

[0214] DDPG method: a single-agent method, which assumes that there is a centrally controlled base station that can observe the global state, and all actions of STs are centrally allocated by the centrally controlled base station.

[0215] 4.2 Simulation results and analysis

[0216] In this section, the method proposed in the present invention is compared with the method in 4.1 through simulation to verify the effectiveness of the resource allocation strategy of the method proposed in the present invention under interference and eavesdropping attacks.

[0217] (1) Analysis of the Convergence of Confidentiality Rate

[0218] Figure 4The convergence analysis of the proposed P-MADDPG method for different learning rates is given. The learning rate is a configurable hyperparameter that can be used to determine the speed at which the model performs resource allocation tasks. First, by fixing the Actor network learning rate to 0.0001, the impact of the Critic network learning rate is analyzed. Figure 4 It can be seen that in the process of resource allocation algorithm training, the learning rate has a great influence on the rapidity and convergence of the system's average confidentiality rate. Although the high learning rate of 0.01 converges faster in the early stage, it causes the average confidentiality rate to fluctuate greatly and become unstable in the later stage. In addition, for the lower learning rate of 0.0001, the proposed method takes a long time to converge and misses the optimal value. As a moderate learning rate, 0.001 has both a faster convergence speed and can converge to the optimal value. Therefore, the learning rate of this section is 0.001.

[0219] (2) Method performance analysis

[0220] like Figure 5 The relationship between the number of training rounds and the average confidentiality rate under the collusion attack considered by the present invention is shown. The results show that when the number of training rounds is less than 50, the proposed P-MADDPG method exhibits an oscillation phenomenon. This is because at the beginning of training, the Critic input of the algorithm is global information, while the input of the I-DDPG method is only the information of the agent n itself. Importantly, as the number of training rounds increases, the proposed method reaches the maximum value and converges after 50 training rounds. Although I-DDPG also reaches the convergence value after 50 training rounds, it misses the global optimal value. The DDPG method reaches convergence after 200 training rounds and is always accompanied by large fluctuations. This is because the DDPG method is easily affected by the inaccuracy caused by the function approximation error, which makes it overfit. Compared with I-DDPG and DDPG, the proposed P-MADDPG-based resource allocation method improves the average confidentiality rate by 33.31% and 43.69% respectively. This is because the proposed method combines the PER technology to prioritize the samples in the experience pool, improves the sample efficiency, and can process high-dimensional action spaces in a shorter time. Therefore, compared with the I-DDPG and DDPG methods, the optimal allocation of transmission power, transmission time and auxiliary interference power can be achieved more quickly.

[0221] Figure 6 The relationship between the average confidentiality rate and the number of training rounds under non-collusion attack is given. Figure 5 and Figure 6The comparison shows that the collusion attack scenarios of interference and eavesdropping considered in the present invention are more damaging to the transmission quality and confidentiality performance of the system. Under a collusion attack, Eves can eliminate the signal of MJ. In addition to reducing the performance of the legitimate link, MJ also promotes eavesdropping by reducing the SINR at the SBS. Under a non-collusion attack, MJ only focuses on reducing the SINR at the SBS, and Eves will also be interfered by MJ. In this case, the eavesdropping performance of Eves will be affected, so the average confidentiality rate will be improved. The results show that in the most serious collusion attack scenario, the P-MADDPG method proposed in the present invention can still show relatively good performance. Next, the performance of the proposed method is further analyzed by giving the influence of different simulation parameters on the average confidentiality rate that can be achieved by the proposed method.

[0222] Figure 7 The impact of various resource allocation methods on the confidentiality rate is evaluated by using different MJ malicious interference powers. from Figure 7 It can be seen that the average confidentiality rate of various schemes decreases with the increase of malicious interference power, and the average confidentiality rate of the system is the highest when there is no interference attack. This is because, according to the definition of confidentiality rate, when the malicious interference power increases, the SINR received at the SBS will be deteriorated, further resulting in a decrease in the achievable rate from ST to SBS, and ultimately resulting in a decrease in the confidentiality rate. The P-MADDPG method proposed in the present invention improves by 14.30% and 34.47% respectively compared with the I-DDPG and DDPG methods when the malicious interference is the most serious, indicating that the proposed method can still find the optimal power and time allocation strategy compared to the comparison algorithm when the attack is aggravated, thereby improving the average confidentiality rate of the network.

[0223] Figure 8 Shows the maximum auxiliary interference power of FJ with different The relationship between the average confidentiality rate and It can be seen that as the maximum auxiliary interference power increases, the average confidentiality rate of the system also increases. This shows that the average confidentiality rate of the network can be improved by introducing auxiliary interference nodes. More importantly, the P-MADDPG method proposed in this invention is superior to other comparison methods. Under the same circumstances, the proposed method can better select the auxiliary interference power.

[0224] Fig. 9 Shows the maximum transmission power P of different STs max The impact on the average minimum confidentiality rate, where P max ={0.12, 0.24, 0.36, 0.48, 0.60, 0.72}. It can be seen that the average confidentiality rate of the system increases with the increase of the maximum transmission power of ST. This is because when P maxWhen the corresponding P n (t) The selected transmit power will also increase, and the SINR received by the SBS will also increase. In addition, it can be seen from Fig. 9 It can be seen that when the transmit power is high, the proposed P-MADDPG-based method can maximize the average confidentiality rate of STs compared with the I-DDPG and DDPG schemes.

[0225] Fig.10 The influence of the number of eavesdroppers Eves on the average confidentiality rate is given. As the number of Eves increases, the average confidentiality rate of the system decreases. n The Eve with the best eavesdropping performance is considered. The increase in the number of Eves will improve its eavesdropping performance, so the average confidentiality rate will show a gradually decreasing trend. The simulation results show that when the number of Eves is 10, the DDPG and I-DDPG methods decrease by 65.2% and 51.97% respectively, while the proposed P-MADDPG-based method decreases by 45.95%, showing better performance than the comparison method. This shows that the proposed method can alleviate the deterioration of the average confidentiality rate of the system under collusion attacks through reasonable resource allocation.

[0226] Fig.11 The performance of the proposed method under different numbers of STs is shown. The results show that in the initial stage, the average confidentiality rate increases with the increase in the number of STs, and gradually reaches a dynamic convergence state in the later stage. This is because, in the initial stage, with the increase in the number of STs, the amount of confidential information received by the SBS will also increase, so the average confidentiality rate will also increase. However, the transmission time slot is fixed, and the increase in STs will lead to a decrease in the transmission time of each ST. The reduction in transmission time slots will also increase the possibility of eavesdropping by eavesdroppers with better channel quality, so the average confidentiality rate will show a convergence trend in the later stage.

[0227] Fig.12 The maximum battery capacity B given by ST max The impact on the proposed resource allocation scheme, where B max ={2,4,6,8,10,12}. As can be seen from the figure, under all resource allocation methods, the average confidentiality rate of ST increases with B. max This is because when B maxWhen it increases, ST can have more energy, so it can transmit at a higher transmission power, and improve the achievable rate of ST at the SBS. In addition, the average confidentiality rate of the P-MADDPG method proposed in the present invention is better than the comparison method. This is because, under the condition of energy limitation, the comparison method cannot reasonably allocate the transmission power and time of each time slot, such as using too much energy to try to transmit in the case of interference, or allocating a small amount of energy for transmission in a channel with better quality, which leads to a decrease in the achievable rate at the SBS. On the contrary, the method proposed in the present invention can ensure the average confidentiality rate under interference and eavesdropping attacks through effective and reasonable energy allocation. On the other hand, Fig.12 The performance of the proposed algorithm in the absence of energy harvesting is also given. max ≤10. Compared with no energy harvesting, ST can use energy harvesting technology to collect the RF energy of PT, MJ, and FJ in the environment. Therefore, it can transmit data to SBS with higher transmission power during the transmission phase, and the average confidentiality rate can be improved. max ≥10, the battery power can meet the transmission of ST in all time slots, so ST will not perform energy harvesting. As a result, the average confidentiality rate converges to the same value under energy harvesting and non-energy harvesting conditions. Therefore, in this model, when energy is limited, the introduction of energy harvesting technology can extend the use time of the device and improve the average confidentiality rate of the system.

[0228] The present invention studies the interference and eavesdropping collusion attacks in energy harvesting cognitive communication networks. Under the scenario of collusion attack, limited user energy and user service quality QoS constraint, ST adopts an interleaved-bottom mixed spectrum access mode and accesses the channel in turn in TMDA mode. By introducing auxiliary interference nodes, degrading the eavesdropping channel, and by reasonably allocating the transmission power, transmission time and auxiliary interference node power, the system average confidentiality rate is maximized. First, according to the network model and the definition of confidentiality rate, the resource optimization expression of the present invention is derived. In view of the optimization problem of resource mutual coupling, unknown attack mode, dynamically changing channel information, and the complex solution of traditional methods, a P-MADDPG continuous action control resource allocation method based on CTDE architecture is proposed. Secondly, in this method, ST and auxiliary interference FJ are jointly modeled as intelligent agents, and the optimal resource allocation strategy is found by designing a reasonable state space, a continuous action space and a reward function to improve the ST network performance. Finally, simulation is carried out. The results show that the average confidentiality rate of the system based on the proposed P-MADDPG method is improved by 33.31% and 43.69% compared with I-DDPG and DDPG, respectively. At the same time, the results also show that the average confidentiality rate of the proposed method is better than that of the comparison method under different parameters. In summary, the above results prove the effectiveness of the proposed resource allocation method in the collusion attack scenario.

[0229] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0230] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0231] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0232] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0233] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0234] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG, characterized in that: include: (1) A multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is proposed; (2) A P-MADDPG resource allocation method based on the Prioritized Experience Replay (PER) mechanism combined with the multi-agent deep deterministic policy gradient (MADDPG) based on the CTDE architecture is proposed to deal with the interference attack and eavesdropping attack of the model in step (1) during the transmission phase and realize cognitive communication resource allocation.

2. According to claim 1, a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG is characterized in that: In step (1), a multi-user distributed energy harvesting cognitive communication network model with coexistence of interference and eavesdropping is considered, and the existence of multiple eavesdroppers and probability-based interference attacks are considered; considering the collusion attack of interference and eavesdropping, the eavesdropping channel is degraded by introducing auxiliary interference nodes, and the DRL technology is used as a means. The goal is to jointly optimize the transmission power, transmission time and auxiliary interference power, and at the same time maximize the optimization problem of the average confidentiality rate of the SUs network under the constraints of the user service quality QoS requirements of the primary user PU network and the secondary user SU network.

3. According to claim 1, a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG is characterized in that: In step (1), the system model of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is constructed as follows: A multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is proposed, which includes three parts: PU network, SU network and malicious attack network. Among them, the PU network contains 1 primary user transmitter PT and 1 primary user receiver PR. The SU network contains N secondary user transmitters ST, 1 secondary user base station SBS and 1 friendly jammer FJ. ST dynamically accesses the PU network spectrum in the underlying or interleaved spectrum access mode and transmits data to the SBS in time division multiple access TDMA mode. FJ degrades the eavesdropping channel by transmitting auxiliary interference noise. It is assumed that FJ generates noise using a pseudo-random sequence known to the legitimate user, which enables the legitimate user to eliminate this noise, while Eve knows the pseudo-random sequence, so Eve cannot eliminate the noise. The malicious attack network contains B malicious eavesdroppers Eve and 1 malicious jammer MJ. Eve eavesdrops on the confidential information of ST received by SBS, and MJ interferes with SBS through attack strategies. Both are committed to reducing the transmission quality and confidentiality performance of the SU network.

4. According to claim 3, a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG is characterized in that: In step (1), the channel gain model of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is constructed as follows: Considering that SBS, ST, FJ, Eve, and MJ are all distributed in the coverage area of ​​the main network, define ST n For the nth ST, ST v For the vth ST, Eve b is the b-th Eve, where n∈{1,2,...,N}, v∈{1,2,...,N} and v≠n, b∈{1,2,...,B}, (x PT ,y PT )、(x PR ,y PR )、(x SBS ,y SBS ) represent the position coordinates of PT, PR and SBS respectively, Respectively represent ST n 、ST v 、Eve b The position coordinates, (x FJ ,y FJ )、(x MJ ,y MJ ) represent the position coordinates of FJ and MJ respectively; the channel model is expressed as follows: Among them, λ is the Rician factor, which is used to describe the signal fading in the wireless channel and represents the difference in signal power between the LOS path and the scattering path; θ:U(0,1) represents the phase of the signal arriving on the LOS path, which takes a value between 0 and 1 and obeys a uniform distribution; CN(·) is a statistical representation, representing a ring-symmetric complex Gaussian random variable; σ 2 =10 -ζ , σ 2 Depends on the path loss, ζ represents a comprehensive path loss indicator; The channel gain model g at time slot t p,n , g n,r , g v,n 、h n,s 、h f,n 、e n,b 、e f,b 、e p,b , g m,s , g m,n Represents PT→ST n 、ST n →PR, ST v →ST n 、ST n →SBS、FJ→ST n 、ST n →Eve b 、FJ→Eve b ,PT→Eve b 、MJ→SBS、MJ→ST n The channel power gain of the eavesdropping channel can be obtained by performing statistical estimation on the information of the eavesdropper channel. Therefore, the channel gain set G(t) perceived by the ST of the entire SU network in time slot t is obtained as follows: Among them, G n (t) is ST n The set of perceived channel gains.

5. According to claim 4, a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG is characterized in that: In step (1), the attack model of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is constructed as follows: Interference model: For the single-channel model, consider the existence of an energy-limited probabilistic interference-based MJ. MJ is the same as ST and has spectrum sensing capability. It considers perfect sensing, but cannot detect the state of ST. It senses the spectrum state before launching an attack. If the spectrum is idle, it launches an interference attack on the SU network with probability β, β∈[0,1]. Use P J (t) represents the interference power in time slot t, Indicates the maximum interference power that MJ can take; if the channel is busy, MJ will remain silent; Eavesdropping model: Each Eve has a radio receiver and always tries to eavesdrop on ST n The data transmitted to the SBS is not eavesdropped by collusion between eavesdroppers, that is, each Eve decodes the confidential message individually, and then considers the Eve with the best receiving performance; the collusion attack between interference and eavesdropping is considered, that is, the eavesdropper Eve also uses the signal processing method to eliminate the interference signal of MJ. In addition, MJ also promotes eavesdropping by reducing the signal-to-noise ratio SINR at the SBS.

6. According to claim 5, a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG is characterized in that: In step (1), the time slot and energy model of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is constructed as follows: 1) Time slot model ST sends data to SBS in TDMA mode, that is, ST sends data in sequence, and the length of each time slot is the same and fixed to T. The time slot T is divided into two stages: Perception stage, ST n PT is sensed at the beginning of the time slot. Considering the case of perfect perception, the ST spectrum sensing set S(t) at time slot t is [I1(t), I2(t), ..., I N (t)] T , where I n (t) is expressed as ST n The perception result, I n (t)∈{0(idle),1(busy)}; the channel occupancy state transition also follows a two-dimensional Markov chain; after sensing, ST n A transmission decision will be made; assuming that the time consumed in the entire stage is τ and the energy consumed is E τ , are all fixed constants; In the transmission phase, after the sensing is completed, ST n Adopting the transmission strategy, the auxiliary jammer FJ sends interference noise to the eavesdropper Eve; considering the transmission mode of mixed bottom layer transmission and interleaved transmission, specifically, when PT occupies the channel, that is, I n (t) = 1, ST n The PU channel is accessed in the bottom layer mode for data transmission. When the PT channel is idle, that is, I n (t) = 0, ST n Access the PU channel in interleaved mode for data transmission; during the transmission phase, ST n , FJ according to the system channel status I n (t), channel gain G n (t) and ST n Battery Level B n (t) Determine the transmission time α n (t), transmission power P n (t) and auxiliary interference power At this time, the transmission time strategy set α(t), transmission power strategy set P(t) and auxiliary interference power strategy set P of STs and FJ are F (t) is expressed as: Among them, α n (t)∈[0,T-τ], and P n (t)∈[0,P max ], P max , ST n , the maximum transmission power of FJ; 2) Energy Model ST has the same structure and function, and has an energy harvesting circuit. All devices except ST are powered by the grid. ST is powered by a rechargeable battery with a capacity of B. max ; Limited by the hardware structure, it follows the mode of acquisition-storage-transmission, that is, the current acquisition needs to be stored in the battery and used in the subsequent transmission stage; define a two-bit variable H n (t) indicates ST n Working mode in time slot t: On this basis, further analysis of ST n The RF energy collected at time slot t When ST v Working in transmission mode, TDMA transmission mode allows ST in the waiting transmission stage to n Energy harvesting; therefore, the above formula also includes ST n Energy collected during the waiting period; the first term in the above formula is ST n The collected ST v and the energy of FJ, the second term is ST n The energy collected in PT or MJ, P T is the PT transmission power, which is set as a fixed constant; define a binary variable J(t)∈{0(interference), 1(sleep)}, indicating whether MJ initiates interference in the current time slot; at this time, the ST energy collection set in the SU network is E(t)=[E1(t),E2(t),...,E N (t)] T , use B n (t) Define the time slot ST n The battery level, B n (t)∈[0,B max ]; Assuming that the battery is ideal, only the sensing and transmission phases consume energy. When the energy harvesting exceeds the maximum battery capacity B max When , the excess energy will overflow, and the battery level at time slot t+1 is: B n (t+1)=min{B max ,B n (t)+E n (t)-P n (t)α n (t)-Z n (t)E τ } Among them, Z n (t) is the spectrum sensing indicator, so the battery state set B(t) of ST at time slot t is: B(t)=[B1(t),...,B2(t),...,B N (t)] T 。 7. According to claim 6, a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG is characterized in that: In step (1), the optimization problem of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is as follows: Considering the operation process of the multi-user distributed energy harvesting cognitive communication network model with coexistence of interference and eavesdropping, the model is analyzed according to the two modes of bottom-layer transmission and interleaved transmission. When PT exists at time slot t, that is, bottom-layer mode transmission, the signal received at the SBS is: Among them, the first item is the ST received by SBS n The transmitted signal, x n (t) represents the transmission signal from ST n ; The second term is the interference signal of PT received by SBS, x p (t) represents the transmission signal to PT; is a product with zero mean and Then SBS receives ST at time slot t n The SINR of a signal is described as: Then ST at time slot t n The achievable rate R at SBS n,sbs (t) is described as: When PT is present, ST n It may interfere with the transmission of PT. In order not to affect the normal transmission of PT, the PU network needs to be analyzed at the same time. The signal received by PR at time slot t is: is a product with zero mean and The SINR received at the PR is: Similarly, now for the eavesdropper Eve b Conduct analysis, Eve b The overheard signal is described as: Among them, the first item is the ST n signal, the second is the PT signal, and the third is the FJ emission interference noise. is a product with zero mean and Variance complex Gaussian noise; then Eve b ST eavesdropped at time slot t n The signal-to-noise ratio SINR is described as: Then ST at time slot t n In Eve b The achievable rate R n,b (t) is described as: According to the definition of confidentiality rate, at this time, SBS in time slot t is about ST n The confidentiality rate is derived as: Where [x] + represents max{x,0}; at the same time, the SINR set at time slot t is obtained as: Similarly, when PT does not exist, that is, ST n When using interleaved mode transmission, SBS and Eve b SINR received at: Define a binary variable J(t)∈{0,1} to indicate whether the SBS is attacked by MJ in time slot t; At this time, the time slot SBS is about ST n The confidentiality rate is derived as: The goal of the optimization problem of the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping is to maximize the average confidentiality rate of the SU network system under interference and eavesdropping attacks by allocating power and time resources. The optimization problem P1 finally established is expressed as: P1: s.t.C1: C2:0≤P n (t)≤P max C3: C4:α n (t)P n (t)≤B n (t)-Z n (t)E τ C5:SINR n,sbs ≥SINR n,sbs,min ,SINR p,r (t)≥SINR p,r,min C6:R n,sec (t)≥R sec,min Among them, L is the number of time slots, C1 is ST n Transmission time constraint; C2 is ST n Transmission power constraint; C3 is FJ interference power constraint; C4 is ST n Battery energy constraint; C5 is the minimum SINR received by PR and SBS p,r,min 、SINR n,sbs,min Constraint; C6 is ST n The confidentiality rate to SBS must be greater than the minimum confidentiality rate R sec,min Require.

8. According to claim 7, a cognitive communication resource allocation method under interference and eavesdropping attacks based on P-MADDPG is characterized in that: The multi-agent deep reinforcement learning (MADRL) mapping is performed on the multi-user distributed energy harvesting cognitive communication network model considering the coexistence of interference and eavesdropping. The specific implementation is as follows: To model the multi-agent and environment, the multi-user resource allocation task is modeled as POMDP. Six-tuple representation; at time slot t, agent n is based on the current state environment Get local observation And according to the strategy π(·|s t ) Execute an action Then get the reward r t n , current status Based on the state transition probability function Transition to next state: Agent: In the energy harvesting cognitive communication network, ST n Together with FJ, they are modeled as one agent, with a total of N agents; State Space The agent needs to know the channel information, including PT state, ST n Initial battery energy, channel gain; In a network where interference and eavesdropping coexist, the state space includes the spectrum sensing set of ST, the channel gain set, the SINR set, and the battery level state of ST; At time slot t, the state space Designed for: Local Observation Space Represents the agent ST n In the local observation space of time slot t, ST n The spectrum sensing set, channel gain set, SINR set and ST n The battery level state composition, local observation space Designed for: Action Space In the resource allocation scenario, agent n allocates transmission time, transmission power, and auxiliary interference to achieve the optimal average confidentiality rate performance under interference and eavesdropping attacks. The action space of agent n is The design is as follows: Reward Space t n :Agent n based on local observations Make a move Then interact with the environment and get rewards; The goal is to maximize the average confidentiality rate of the entire SU network under the relevant constraints, so the reward function must include R n,sec (t), the larger the reward function, the greater the R n,sec The larger (t) is, the better the action decision made by the agent will be. Considering the optimization problem, the penalty term is introduced at the same time. The reward function r of agent n is t n The design is as follows: in, ρ and χ represent penalty values. Introducing penalty terms can stimulate the agent to better learn unknown environments.

9. The method for cognitive communication resource allocation under interference and eavesdropping attacks based on P-MADDPG according to claim 8, characterized in that: In step (2), the P-MADDPG resource allocation method based on the combination of the Prioritized Experience Replay PER mechanism and the multi-agent reinforcement learning algorithm MADDPG based on the CTDE architecture is specifically implemented as follows: The agent set is represented as Agent n is composed of 4 neural networks, including target policy network, online policy network, target Q network and online Q network; target policy network and online policy network belong to Actor network, target Q network and online Q network belong to Critic network; in addition, there is an experience replay pool Used to store the experience transformation tuples in the training phase; define μ = {μ1,μ2,...,μ N } is the set of deterministic strategies of all agents in the online strategy network, μ′={μ1′,μ2′,...,μ′ N } is the set of deterministic strategies in the target strategy network, and θ={θ1,θ2,...,θ N } represents the parameters of the deterministic strategy μ, θ′={θ1′,θ2′,...,θ′ N } is the parameter of the deterministic strategy μ′; the online strategy network of agent n at time slot t is determined by the deterministic strategy μ n The local observation Mapping to specific actions Right now In order to explore the optimal action strategy, random noise is added to the action strategy, so the actual action description is: Agent n follows the action strategy Interact with the environment to obtain r t n , next observation The agent stores its own experience tuples in the replay pool middle, The training experience tuple (s t ,a t ,r t ,s t+1 ) contains the current state, action, reward and next state of all agents; the Critic network Perform action value evaluation and calculate the centralized action-value function Q(s) by inputting state and action t ,a t ), where the states and actions refer to the states and actions of all agents, i.e. (s t ,a t ); The centralized action-value Q function is specifically calculated by the Bellman equation: Where γ is the discount factor, Q ω (s,a) is used to evaluate the current state s t Take action a t The effect is to help the online policy network of agent n improve its policy; PER sets a priority for each experience tuple based on TD-error. The experience tuple with a larger TD-error has a greater probability of being sampled. Use δ n represents its TD-error, δ n Calculated as: Among them, ω n ,ω n ′ represents the online critic network and target critic network parameters of agent n, and the experience tuple x n The priority is calculated by the following formula: p n =|δ n |+g where ζ is a small positive constant, ensuring that δ n It can also be sampled normally when it is 0; during the training phase, the network parameters of Actor and Critic are obtained by replaying the experience pool. The sampling batch size using PER technology is The experience tuple is used to update, and this process will continue until the end; x n The probability of being selected P(x n )With the priority monotony, P(x n ) is described as: Among them, α1∈[0,1] is used to adjust the importance of the priority in the sampling process. α1=0 corresponds to the original uniform random sampling, and α1=1 corresponds to the direct use of the TD-error value. In order to alleviate the problem of oscillation in the training phase caused by changes in the sample sampling frequency due to PER, the importance sampling weight is introduced: represents the capacity of the experience replay pool, and α2 is the degree of error correction; then, the Critic network learns the action value Q function corresponding to the optimal strategy by minimizing the loss. The Critic network loss is described as: y is the sum of the immediate reward and the discounted target Q value obtained by the target Q network: in, For ST n The target strategy network outputs actions; the Actor network of each agent updates the parameters θ and adjusts the strategy according to the centralized action-value Q function calculated by the Critic network and the observation information to optimize the objective function J(θ n ), the optimal strategy of agent n is expressed as: Parameter θ n Directly in the Actor network Deterministic policy gradient direction adjustment: It consists of two parts: the gradient of the action value Q function from the Critic network and its own deterministic policy action gradient To improve the stability of learning, as the online policy network and the online Q network are updated in real time, the target policy network parameter θ n ′ and the parameter ω of the target Q network n ′ will be updated by the soft update rule: Among them, ξ a ∈[0,1],ξ c ∈[0,1] is the soft update factor of the Actor and Critic networks; after training, agent n can choose its own actions based on local observations without knowing the information of other agents.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 9 can be implemented.

Images