Double-agent double-delay depth deterministic strategy gradient optimization and secure transmission method and system

Abstract

The invention provides the technical field of wireless communication and information security, particularly relates to a double-agent double-delay depth deterministic policy gradient optimization and secure transmission method and system based on deep reinforcement learning and cooperative interference under the assistance of an intelligent reconfigurable surface, and is applied to the technical field of wireless communication and information security. Double agents are adopted for deep reinforcement learning to obtain a beam matrix optimal strategy in wireless communication, and under the assistance of a cooperative jammer and an intelligent reconfigurable surface, an interference signal transmitted by the cooperative jammer is reflected to an eavesdropper, so that the eavesdropping rate can be reduced, the communication secrecy rate is improved, and the communication efficiency is improved. The communication quality can be improved by reducing the influence of interference signals on legal users, and the secrecy rate of the system is improved.

Description

Technical Field

[0001] The present application relates to the fields of wireless communications and information security technologies, and specifically to a dual-agent dual-delay deep deterministic policy gradient optimization and secure transmission method and system based on deep reinforcement learning and collaborative interference with the assistance of an intelligent reconfigurable surface. Background Art

[0002] With the breakthrough of wireless communication technology, the speed of laying 5G wireless communication network is gradually accelerating, and the number of network access points has been greatly improved. According to Cisco's latest annual Internet report white paper, the number of mobile Internet devices is growing rapidly. The report mentioned that in 2023, the number of people with mobile Internet devices will account for 70% of the world's total population, and the number of mobile Internet users will reach 5.7 billion, an increase of 600 million compared with 2018. According to forecasts, the scale of global Internet traffic data will continue to increase at a high speed. The annual average growth rate of traffic data will reach 55% between 2020 and 2030, and its final scale will reach 501.6 billion bytes.

[0003] Since the wireless transmission environment has its own inherent characteristics, such as broadcast characteristics and open wireless network access, it is very easy to be eavesdropped and interfered by illegal devices. In addition, a large amount of information containing user privacy is now transmitted in wireless networks, and they all face the risk of being eavesdropped by illegal devices.

[0004] Therefore, ensuring the secure communication transmission of legitimate users in wireless communication networks is one of the tasks that needs to be solved urgently. Summary of the invention

[0005] In view of this, the present application provides a dual-agent dual-delay deep deterministic policy gradient optimization and secure transmission method and system based on deep reinforcement learning and cooperative interference with the assistance of an intelligent reconfigurable surface. With the assistance of an intelligent reconfigurable surface (IRS), during the communication process, the eavesdropping rate can be reduced by reflecting interference signals to the eavesdropper, and / or the signal-to-interference-plus-noise ratio (SINR) of the legitimate user can be improved by assisting the communication between the legitimate user and the base station, thereby realizing secure transmission of wireless communication based on deep reinforcement learning and cooperative interference.

[0006] This application provides the following technical solutions:

[0007] The present application provides a dual-agent dual-delay deep deterministic policy gradient optimization method, comprising:

[0008] Through the continuous interaction between the first agent and the environment, the first beamforming matrix W of the base station is learned B and the optimization strategy of the second reflection matrix Ψ of the intelligent reconfigurable surface; wherein the comprehensive channel gain H Cand the comprehensive channel damage D C As the first state space s n,1 For the first beamforming matrix W B And the second reflection matrix Ψ is optimized and learned; the comprehensive channel gain H C It includes the comprehensive channel state information from the base station to the eavesdropper, the intelligent reconfigurable surface, and the legitimate user, as well as the comprehensive channel state information from the intelligent reconfigurable surface to the legitimate user and the eavesdropper; the comprehensive channel damage D C including comprehensive channel state information from cooperative jammers to smart reconfigurable surfaces, legitimate users, and eavesdroppers;

[0009] Through the continuous interaction between the second agent and the environment, the third beamforming matrix W of the cooperative interferer is learned J The optimization strategy of C As the second state space s n,2 The third beamforming matrix W J Conduct optimized learning;

[0010] Among them, both the first agent and the second agent adopt a double-delay deep deterministic policy gradient algorithm network model.

[0011] Preferably, the first beamforming matrix W B , the second reflection matrix ψ, the third beamforming matrix W J The optimal strategy is to maximize the confidentiality energy efficiency problem as the optimization goal and optimize the learning to obtain:

[0012]

[0013] stC1:||W B || 2 ≤P max ,||W J || 2 ≤P max ,

[0014]

[0015] Among them, constraint C1 stipulates that the actual transmission power of the cooperative interferer and the base station is lower than the set maximum transmission power; the threshold is the minimum confidentiality rate required for legitimate users to decode; constraint C2 specifies the minimum confidentiality rate at which each legitimate user can correctly decode information; constraint C3 is the constraint on the phase shift of the intelligent reconfigurable surface;

[0016] SEE stands for System Confidentiality Energy Efficiency:

[0017]

[0018] P total The total energy consumption of the system is:

[0019]

[0020] Where P Bs , P CJ and P IRS denote the energy consumed by the base station, cooperative jammer, and smart reconfigurable surface, respectively;

[0021] Represents the confidentiality rate of the kth user:

[0022]

[0023] The achievable rate of the kth legal user is defined as:

[0024]

[0025] When the e-th eavesdropper eavesdrops on the signal of the k-th legitimate user, the achievable rate of the e-th eavesdropper is:

[0026]

[0027] γ k It represents the signal-to-interference-noise ratio of the kth legitimate user:

[0028]

[0029] n k represents Gaussian white noise with mean 0, and the sets of legitimate users and eavesdroppers are defined as and ε={1,2,…,E}.

[0030] Preferably, the reward function includes a penalty for violations of system constraints when interacting with the environment, wherein the reward function is set as:

[0031] r n =tanh(SEE-λ1μ1-λ2μ2-λ3μ3)

[0032] Among them, μ1, μ2 and μ3 are the penalties corresponding to the three constraints in the optimization objective; The weights used to guide the agent to achieve a preset convergence.

[0033] Preferably, the action space a of the first agent n,1 The first beamforming matrix W B and the second reflection matrix Ψ, the action space a of the second agent n,2 The third beamforming matrix W JThe first reward function of the first agent and the second reward function of the second agent are set to the same reward function.

[0034] Preferably, the channels are all Rician fading channel models, and the fading of the channels is calculated as follows:

[0035]

[0036] Among them, PL d is the path loss corresponding to the communication distance d, let PL d =-PL0-10α i,j log 10 (d / D0), d is the communication distance, D0 is the reference distance and its value is 1 meter, and the path loss PL0 corresponding to the reference distance is taken as 30dB; α i,j is the path loss factor, where i∈{B,J,I} and j∈{E,U k}; G LoS and G NLoS represent the line-of-sight LoS path and the Rayleigh fading component respectively; β is the Rayleigh attenuation factor.

[0037] Preferably, in any one of the dual-agent dual-delay deep deterministic policy gradient optimization methods described in the present application, four fully connected hidden layers and four fully connected hidden layers are deployed in the Actor network and the Critic network of the first agent and the second agent;

[0038] and / or, the number of neurons in the four fully connected hidden layers of the first agent is [800, 600, 512, 256], and the number of neurons in the four fully connected hidden layers of the second agent is [400, 300, 256, 128];

[0039] And / or, the learning rates of the Actor network and the Critic network of the first agent and the second agent are set to 0.0001 and 0.001, respectively.

[0040] Preferably, the first agent and the second agent are trained for 300 rounds, with 100 steps in each round.

[0041] The present application also provides a wireless communication security transmission method, which is applied to the communication process between a legitimate user and a base station, and the wireless communication security transmission method includes:

[0042] The base station uses a first beamforming matrix W B The optimization strategy of transmitting communication signals into space;

[0043] The third beamforming matrix W is obtained by the cooperative interferer. J The optimization strategy of transmitting interference signals to the eavesdropper to reduce the eavesdropper's eavesdropping rate;

[0044] And, through the intelligent reconfigurable surface, according to the optimization strategy of the second reflection matrix Ψ, the interference signal is reflected to the eavesdropper to reduce the eavesdropping rate, and / or the useful signal is reflected to the legitimate user to assist the communication between the legitimate user and the base station, so as to improve the signal-to-interference-noise ratio of the legitimate user;

[0045] Among them, the first beamforming matrix W B , the second reflection matrix Ψ, the third beamforming matrix W J The optimization strategy adopts the optimization strategy obtained by the dual-agent dual-delay deep deterministic policy gradient optimization method as described in any one of the present applications.

[0046] The present application also provides a dual-agent dual-delay deep deterministic policy gradient optimization system, including: a first agent and a second agent;

[0047] The first agent is used to continuously interact with the environment and learn the first beamforming matrix W of the base station. B and the optimization strategy of the second reflection matrix Ψ of the intelligent reconfigurable surface; wherein the comprehensive channel gain H C and the comprehensive channel damage D C As the first state space s n,1 For the first beamforming matrix W B And the second reflection matrix Ψ is optimized and learned; the comprehensive channel gain H C It includes the comprehensive channel state information from the base station to the eavesdropper, the intelligent reconfigurable surface, and the legitimate user, as well as the comprehensive channel state information from the intelligent reconfigurable surface to the legitimate user and the eavesdropper; the comprehensive channel damage D C including comprehensive channel state information from cooperative jammers to smart reconfigurable surfaces, legitimate users, and eavesdroppers;

[0048] The second agent is used to continuously interact with the environment and learn the third beamforming matrix W of the cooperative interferer. J The optimization strategy of C As the second state space s n,2 The third beamforming matrix W J Optimize learning.

[0049] The present application also provides a wireless communication security transmission system, comprising: at least one intelligent reconfigurable surface, at least one base station, at least one cooperative jammer, at least one eavesdropper and at least one legitimate user;

[0050] The intelligent reconfigurable surface is equipped with multiple reflective surfaces; the intelligent reconfigurable surface, the base station, and the cooperative jammer are all equipped with multiple antennas, and the eavesdropper and the legitimate user are equipped with a single antenna; the base station is used to transmit the signal according to the first beamforming matrix WB The communication signal is transmitted to the space according to the optimization strategy of the third beamforming matrix W; the eavesdropper is used to eavesdrop when the legitimate user communicates with the base station; the cooperative interferer is used to J The intelligent reconfigurable surface is used to reflect the interference signal to the eavesdropper to reduce the eavesdropping rate of the eavesdropper according to the optimization strategy of the second reflection matrix Ψ during the communication between the legitimate user and the base station, and / or reflect the useful signal to the legitimate user to assist the communication between the legitimate user and the base station, so as to improve the signal-to-interference-noise ratio of the legitimate user;

[0051] Among them, the first beamforming matrix W B , the second reflection matrix Ψ, the third beamforming matrix W J The optimization strategy adopts the optimization strategy obtained by the dual-agent dual-delay deep deterministic policy gradient optimization method as described in any embodiment of the present application.

[0052] The present application also provides a computing device, the computing device comprising:

[0053] at least one processor, memory, and input-output unit;

[0054] In which, the memory is used to store computer programs, and the processor is used to call the computer programs stored in the memory to execute: the steps of the dual-agent dual-delay deep deterministic policy gradient optimization method as described in any one of the present applications; or the steps of the wireless communication secure transmission method as described in the present application.

[0055] The present application also provides a computer-readable storage medium, which includes instructions, which, when executed on a computer, enable the computer to execute: the steps of the dual-agent dual-delay deep deterministic policy gradient optimization method as described in any one of the present application; or the steps of the wireless communication secure transmission method as described in the present application.

[0056] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above technical solutions adopted in the present application include at least:

[0057] (1) More suitable for power-sensitive devices

[0058] This application introduces a cooperative jammer, which will increase the system's additional power burden, but the purpose is to enable it to send noise to the eavesdropper to reduce the eavesdropper's confidentiality rate. Therefore, this application is different from other existing technical solutions in measuring system indicators. Compared with improving the system confidentiality rate, improving the system confidentiality energy efficiency has a wider range of application and is more scientific.

[0059] (2) Improved security of wireless communications

[0060] In this application, the system can not only improve the confidentiality rate of the system through cooperative jammers, but also improve the communication quality between legitimate users and base stations through IRS, and can also reflect the interference signal emitted by the cooperative jammers to the eavesdroppers, thereby further improving the confidentiality rate of the system and achieving a higher security level of wireless communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 It is a structural diagram of a wireless communication security transmission system in this application;

[0063] Figure 2 It is a schematic diagram of the network model structure of a dual-agent dual-delay deep deterministic policy gradient optimization algorithm (DTD3) in this application;

[0064] Figure 3 It is a simulation comparison diagram of DTD3 and traditional TD3 in this application;

[0065] Figure 4 is the structure of the computer storage medium in this application;

[0066] Figure 5 It is a structural diagram of a computing device in this application. DETAILED DESCRIPTION

[0067] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0068] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0069] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0070] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0071] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the examples can be practiced without these specific details.

[0072] With the rapid development of mobile Internet and artificial intelligence, traditional 5G wireless network communication technology may not be able to meet people's real needs for future networks.

[0073] At present, researchers and technical experts have conceived and designed future networks, such as proposing super 5G (B5G) wireless communication networks or 6G wireless communication networks. Although super 5G or 6G wireless networks are based on the technical architecture of modern 5G, combined with people's future needs for wireless networks, higher technical requirements are proposed, such as ultra-high data transmission rate, such as a peak rate of 1Tb per second; excellent energy consumption ratio, such as supporting tiny IoT devices; high reliability and low latency, such as achieving end-to-end latency of less than 1 millisecond. However, whether it is the current 5G technology or the future B5G / 6G wireless network communication technology, the goal of continuously increasing network capacity and the number of accessible devices is unchanged. In addition, due to their different architectures, the devices accessing the wireless network have different levels of security protection. Combined with the characteristics of the wireless transmission environment, the security performance of the wireless transmission network is facing a severe test, especially now that a large amount of information containing user privacy is transmitted in the wireless network, they are all facing the risk of being eavesdropped by illegal devices.

[0074] Traditional security solutions have certain computing requirements for both the receiving and sending ends. For mobile application networks that already require low energy consumption, such as small IoT devices, excessive computing means increased energy consumption, which is unacceptable.

[0075] In addition, in terms of physical layer security technology, some characteristics of the channel such as fading, interference and noise can be used to achieve confidential transmission. For example, the currently proposed intelligent reflecting surface (IRS) can program the wireless environment. The organic combination of the two will greatly reduce the security threats faced by wireless communications. IRS can program the wireless communication environment, breaking the limited assumption that the wireless communication environment is an uncontrollable factor, and is considered to be an important prerequisite technology for the realization of 6G wireless communication networks.

[0076] In addition, artificial intelligence technology performs better than traditional methods in dealing with problems with large parameters and is more efficient in dealing with non-convex problems. Artificial intelligence algorithms can be deployed at base stations to control IRS.

[0077] Therefore, it is urgent to invent a method that utilizes the characteristics of IRS itself and can be combined with other security technologies to improve the system confidentiality rate while considering the confidentiality energy efficiency. This has extremely high research value and significance for both existing wireless communication networks and the next generation of wireless communications and the security performance of wireless communications.

[0078] Based on this, this application proposes a new solution for secure transmission based on deep reinforcement learning and cooperative interference intelligent reconfigurable surface: Figure 1 As shown, in the wireless communication system, there is at least one IRS (intelligent reconfigurable surface), one BS (base station), one cooperative jammer (CJ, also called cooperative jammer), one or more eavesdroppers, and one or more legal users. Since the eavesdropper will eavesdrop when the legal user is communicating with the BS, the CJ is used to transmit an interference signal to the eavesdropper to reduce the eavesdropping rate of the eavesdropper, and the IRS can be used to reduce the eavesdropping rate by reflecting the interference signal to the eavesdropper during the communication process, and / or assist the communication between the legal user and the base station to improve the signal-to-interference-plus-noise ratio (SINR) of the legal user.

[0079] The technical solutions provided by various embodiments of the present application are described below in conjunction with the accompanying drawings.

[0080] like Figure 1 As shown, there are five roles in this communication system, where the BS has M antennas for communicating with K legitimate users, and each legitimate user and eavesdropper is equipped with an antenna. The set of legitimate users and eavesdroppers is defined as ε={1,2,…,E}. The cooperative jammer sends jamming signals to the eavesdropper to improve the security of the system. Its number of antennas is the same as that of the BS, which is M. The IRS is equipped with N reflection units, which have the following two functions: (1) It can reflect the jamming signals emitted by the cooperative jammer to the eavesdropper; (2) It can reduce the impact of the jamming signals on legitimate users. The above two points can improve the confidentiality rate of the system. The reflection beamforming matrix of IRS is expressed as: in, ψ n Phase shift for elements

[0081] refer to Figure 1 In the communication process between a legitimate user and a base station, the wireless communication security transmission method provided by the present application may include:

[0082] At the base station (BS) side, according to the first beamforming matrix W of the BS B The optimization strategy of transmitting communication signals into space;

[0083] On the cooperative jammer (CJ) side, according to the third beamforming matrix W of CJ J The optimization strategy of transmitting interference signals to the eavesdropper to reduce the eavesdropper's eavesdropping rate;

[0084] And, on one side of the intelligent reconfigurable surface (IRS), according to the optimization strategy of the second reflection matrix Ψ, the interference signal is reflected to the eavesdropper to reduce the eavesdropping rate, and / or the useful signal is reflected to the legitimate user to assist the communication between the legitimate user and the base station, so as to improve the signal-to-interference-noise ratio of the legitimate user.

[0085] It should be noted that the communication process can also be carried out in other ways. It is only necessary to use the optimal strategy in the communication process to reduce the eavesdropper's eavesdropping rate and / or improve the communication quality of legitimate users, so this application will no longer elaborate on the communication process.

[0086] The following is based on the dual-agent dual-delay deep deterministic policy gradient optimization method (DTD3 algorithm for short) to illustrate the optimization learning of the intelligent reconfigurable surface based on deep reinforcement learning and cooperative interference to obtain the optimal strategy, that is, the first beamforming matrix W B , the second reflection matrix ψ, the third beamforming matrix W J Optimization strategy.

[0087] For the kth legal user, the received signal can be expressed as:

[0088]

[0089] in, represents the data sent by BS to the kth user, sJ,k Indicates an interfering signal. So w B,k is the BS beamforming matrix composed of beamforming vectors. So w J,k The interference vectors form a cooperative interference beamforming matrix. k Represents Gaussian white noise (AWGN) with a mean of zero. Definition is the integrated channel gain between the BS and all receivers, defined as As the combined channel impairments between the cooperative jammer and all receivers.

[0090] First, the SINR of the kth user is expressed as

[0091]

[0092] For the kth user, the achievable rate is defined as

[0093]

[0094] If the e-th eavesdropper eavesdrops on the signal of the k-th user, the achievable rate of the e-th eavesdropper is

[0095]

[0096] Finally we get the confidentiality rate of the kth user:

[0097]

[0098] Where [a] + =max(a,0).

[0099] It should be noted that the introduction of CJ to improve the system's SEE is not without cost. In addition to interfering with legitimate users, CJ will also increase the system's energy consumption. However, as long as it is optimized properly, the introduction of CJ will have more benefits than disadvantages.

[0100] Therefore, when calculating SEE, the energy consumption model of the entire system is established. The total energy consumption of the system (abbreviated as total energy consumption) is defined as

[0101]

[0102] This value is calculated by the sum of the energy consumed by BS, CJ and IRS, respectively. BS , P CJ and P IRS express.

[0103] Secondly, the system confidentiality energy efficiency (abbreviated as confidentiality energy efficiency) can be obtained by combining the total power and total confidentiality of the system:

[0104]

[0105] At this point, the optimization learning is carried out with maximizing the confidentiality energy efficiency as the priority goal. The optimization goal can be expressed as:

[0106]

[0107] stC1:||W B || 2 ≤P max ,||W J || 2 ≤P max ,

[0108]

[0109] Constraint C1 stipulates that the actual transmit power of CJ and BS must be lower than the set maximum transmit power. is the minimum confidentiality rate required for legitimate users to decode. Constraint C2 ensures the minimum confidentiality rate that every legitimate user can correctly decode the information. C3 is the constraint on the IRS phase shift.

[0110] In addition, since the beamforming matrix W of CJ J It has a great impact on the SEE of the system, which can increase or reduce the confidentiality rate of the system. Therefore, it is difficult for traditional TD3 optimization learning (i.e., a single agent) to optimize these variables at the same time. This application proposes a dual-agent dual-delay deep deterministic policy gradient optimization method (abbreviated as DTD3 algorithm) based on dual-agent optimization learning.

[0111] refer to Figure 2 As shown in the figure, the present invention constructs two TD3 agents to optimize these variables respectively, instead of using a single agent based on the traditional DRL network. By initializing the two TD3 agents respectively, let the first TD3 agent (ie, TD3 Agent 1) have a beamforming matrix W for the BS. B and the IRS reflection matrix Ψ, the second TD3 agent (i.e., TD3 Agent 2) mainly optimizes the CJ beamforming matrix W J Optimization is performed, so that the final optimization strategy can be obtained after multiple rounds of training of DTD3 based on the continuous interaction between the two agents and the environment.

[0112] exist Figure 2 The network architecture of DTD3 is shown in Figure 1. The first TD3 agent will be H C and D C As the state of W B and Ψ, while another TD3 agent optimizes DC As the state of W J Both agents use the same reward function and they combine the actions they take to interact with the environment.

[0113] The following is a brief introduction to the network architecture of the DTD3 algorithm.

[0114] The first TD3 network can slowly accumulate experience through continuous interaction with the environment, and then learn the reflection matrix Ψ of the IRS and the beamforming matrix W of the BS. B The optimization strategy of . In which, the state space s of the first TD3 agent is n,1 Assume that it includes the comprehensive CSI from BS to eavesdroppers, IRS, legitimate users, and . In addition, it also includes the CSI from IRS to legitimate users and eavesdroppers, so these CSIs are defined as H C , and the CSI from the cooperative jammer to the IRS, legitimate users, and eavesdroppers, defined as D C .

[0115] Also, the action space a of the first TD3 agent n,1 By the matrix Ψ and W B Composition, set the reward function of TD3 agent 1 to r n =tanh(SEE-λ1μ1-λ2μ2-λ3μ3), this reward function includes penalties for behaviors that violate the system constraints when interacting with the environment. These penalties are represented by μ1, μ2, and μ3, which correspond to the three constraints in the optimization objective. In addition, the weight λ i , The introduction of can more effectively guide the agent to achieve better convergence. Usually, this complex matrix is ​​divided into two parts: real part and imaginary part, such as W B =Re{W B}+Im{W B}, ensuring that these matrices can be processed and operated correctly.

[0116] And, in the first TD3 agent, optimize W B While optimizing W J Now let’s consider the components of the second TD3 agent, such as the state s n,2 、Action a n,2 and reward r n Its state space s n,2 By D C Composition, action space a n,2 The active beamforming matrix W of CJ J .

[0117] In some examples, the reward function r of the second TD3 agent is n It can be the same as the first TD3 agent. Also, both agents stop training after reaching a certain number of iterations.

[0118] Next, the present invention is further illustrated by the following simulation example.

[0119] Assume K = 2, E = 2, M = 4, N = 16, set the coordinates of BS, IRS, cooperative jammer, two eavesdroppers and two users to [0, 0, 0], [0, 10, 20], [40, 0, 0], [40, 30, 0], [50, 30, 0], [0, 20, 0], [10, 30, 0], respectively, and set PL = -PL0-10α i,j log 10 (d / D0)dB is used as the large-scale fading model of the system, where d is the communication distance, D0 is the reference distance (for example, 1 meter), and the path loss PL0 (for example, 30dB). i,j is the path loss factor, where i∈{B,J,I} and j∈{E,U k The path loss factors from BS to the kth user, cooperative jammer to eavesdropper, BS to eavesdropper, CJ to the kth user, BS to IRS, IRS to the kth user, cooperative jammer to IRS, and IRS to eavesdropper can be set to be the same or different. Here they are set to be the same:

[0120] Also, for small-scale fading, assuming that all channels involved are Rician fading channel models, the fading is calculated as follows:

[0121]

[0122] Among them, G LoS and G NLoS represent the line-of-sight (LoS) path and the Rayleigh fading component respectively. β is the Rayleigh attenuation factor.

[0123] In addition, in the first TD3 agent, four fully connected hidden layers are deployed in both the Actor network and the Critic network, with the number of neurons being [800, 600, 512, 256], and the learning rates of the Actor and Critic networks are set to 0.0001 and 0.001, respectively. The network structure of the second TD3 agent can be the same as that of the first TD3 agent, with the number of neurons in the four layers being different, set to [400, 300, 256, 128].

[0124] DTD3 was trained for 300 rounds, with 100 steps per round, and other parameters were set to PBS =P CJ =10dBm, P IRS =0dBm, P max =10dBm and n k =n e =-80dBm, for this application and the traditional solution, Figure 3 The simulation results are compared and analyzed to illustrate the SEE of each method. The curve corresponding to the scheme of the present application is labeled "DTD3-CJ", the first scheme in the traditional scheme is a TD3 network with CJ, and its benchmark curve is labeled "TD3-CJ", and the second scheme is a TD3 network without CJ, and its benchmark curve is labeled "TD3-noCJ".

[0125] like Figure 3 As shown in the figure, it is obvious that the performance of SEE using the "DTD3-CJ" method is the best. The reason why the "TD3-noCJ" method without CJ has a poor performance is that it only considers the maximization of the SINR of the receiving user and can only reduce the eavesdropper's eavesdropping rate by adjusting the reflective elements in the IRS.

[0126] As in each embodiment of the present application, the "DTD3-CJ" method provided in the present application will choose how to maximize SEE according to the actual situation. B If the gain provided by the reflective elements is too small, it will try to increase the SEE of the system by allocating more power to CJ.

[0127] Based on the same inventive concept, the present application also provides a dual-agent dual-delay deep deterministic policy gradient optimization system, referring to Figure 2 In the embodiment, the system may include two agents, namely a first agent and a second agent; wherein the first agent is used to continuously interact with the environment and learn the first beamforming matrix W of the base station. B and the optimization strategy of the second reflection matrix Ψ of the intelligent reconfigurable surface; wherein the comprehensive channel gain H C and the comprehensive channel damage D C As the first state space s n,1 For the first beamforming matrix W B And the second reflection matrix Ψ is optimized and learned; the comprehensive channel gain H C It includes the comprehensive channel state information from the base station to the eavesdropper, the intelligent reconfigurable surface, and the legitimate user, as well as the comprehensive channel state information from the intelligent reconfigurable surface to the legitimate user and the eavesdropper; the comprehensive channel damage D C It includes comprehensive channel state information from cooperative jammers to intelligent reconfigurable surfaces, legitimate users, and eavesdroppers; the second agent is used to continuously interact with the environment and learn the third beamforming matrix W of the cooperative jammer.J The optimization strategy of C As the second state space s n,2 The third beamforming matrix W J Optimize learning.

[0128] Exemplary Media

[0129] After introducing the method and system of the exemplary embodiment of the present invention, next, reference is made to Figure 4 A computer-readable storage medium according to an exemplary embodiment of the present invention is described.

[0130] Please refer to Figure 4 , the computer-readable storage medium shown is a CD 80, on which a computer program (i.e., a program product) is stored. When the computer program is executed by the processor, each step recorded in the above method implementation will be implemented. The specific implementation of each step in the method example can be referred to the above example, and will not be repeated here.

[0131] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0132] In a possible implementation, the present invention may also provide a form of a program product that implements data processing, which includes program code. When the program product runs on a terminal device, the program code is used to enable the terminal device to execute several steps of the method described in any of the aforementioned embodiments.

[0133] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on a user device, partially on a user device, as an independent software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0134] Exemplary Computing Devices

[0135] After introducing the method, system and medium of the exemplary embodiment of the present invention, next, reference is made to Figure 5 A computing device for processing an exemplary embodiment of the present invention.

[0136] Figure 5A block diagram of an exemplary computing device 90, which may be a computer system or a server, is shown that is suitable for implementing embodiments of the present invention. Figure 5 The computing device 90 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present invention.

[0137] like Figure 5 As shown, the components of the computing device 90 may include, but are not limited to: one or more processors or processing units 901, a system memory 902, and a bus 903 connecting different system components (including the system memory 902 and the processing unit 901).

[0138] The computing device 90 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device 90, including volatile and non-volatile media, removable and non-removable media.

[0139] The system memory 902 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 9021 and / or cache memory 9022. The computing device 90 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the ROM 9023 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 is not shown in the Figure 5 As shown in FIG. 1 , a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 903 via one or more data medium interfaces. System memory 902 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.

[0140] A program / utility 9025 having a set (at least one) of program modules 9024 may be stored, for example, in system memory 902, and such program modules 9024 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 9024 generally perform the functions and / or methods of the embodiments described herein.

[0141] The computing device 90 may also communicate with one or more external devices 904 (e.g., a keyboard, a pointing device, a display, etc.). Such communication may be performed via an input / output (I / O) interface 905. Furthermore, the computing device 90 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 906. Figure 5 As shown, the network adapter 906 communicates with other modules (such as the processing unit 901, etc.) of the computing device 90 via the bus 903. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with computing device 90 .

[0142] The processing unit 901 executes various functional applications and data processing by running the program stored in the system memory 902. The specific implementation of each step is not repeated here. It should be noted that although several units / modules or sub-units / sub-modules of the model processing device are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be concretized in one unit / module. On the contrary, the features and functions of one unit / module described above can be further divided into multiple units / modules to be concretized.

[0143] In this specification, the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments described later, the description is relatively simple, and the relevant parts can be referred to the partial description of the previous embodiments.

[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A dual-agent dual-delay deep deterministic policy gradient optimization method, characterized in that: include: Through the continuous interaction between the first agent and the environment, the first beamforming matrix W of the base station is learned B and the optimization strategy of the second reflection matrix Ψ of the intelligent reconfigurable surface; wherein the comprehensive channel gain H C and the comprehensive channel damage D C As the first state space s n,1 For the first beamforming matrix W B And the second reflection matrix Ψ is optimized and learned; the comprehensive channel gain H C It includes the comprehensive channel state information from the base station to the eavesdropper, the intelligent reconfigurable surface, and the legitimate user, as well as the comprehensive channel state information from the intelligent reconfigurable surface to the legitimate user and the eavesdropper; the comprehensive channel damage D C including comprehensive channel state information from cooperative jammers to smart reconfigurable surfaces, legitimate users, and eavesdroppers; Through the continuous interaction between the second agent and the environment, the third beamforming matrix W of the cooperative interferer is learned J The optimization strategy of C As the second state space s n,2 The third beamforming matrix W J Conduct optimized learning; Among them, both the first agent and the second agent adopt a double-delay deep deterministic policy gradient algorithm network model.

2. The dual-agent dual-delay deep deterministic policy gradient optimization method according to claim 1, characterized in that: The first beamforming matrix W B , the second reflection matrix Ψ, the third beamforming matrix W J The optimal strategy is to maximize the confidentiality energy efficiency problem as the optimization goal and optimize the learning to obtain: Among them, constraint C1 stipulates that the actual transmission power of the cooperative interferer and the base station is lower than the set maximum transmission power; the threshold is the minimum confidentiality rate required for legitimate users to decode; constraint C2 stipulates the minimum confidentiality rate for each legitimate user to correctly decode the information; constraint C3 is the constraint of the phase shift of the intelligent reconfigurable surface; SEE stands for System Confidential Energy Efficiency: P total The total energy consumption of the system is: Where P BS , P CJ and P IRS denote the energy consumed by the base station, cooperative jammer, and smart reconfigurable surface, respectively; represents the confidentiality rate of the kth user: The achievable rate of the kth legal user is defined as: When the e-th eavesdropper eavesdrops on the signal of the k-th legitimate user, the achievable rate of the e-th eavesdropper is: γ k It represents the signal-to-interference-noise ratio of the kth legitimate user: n k represents Gaussian white noise with mean 0, and the sets of legitimate users and eavesdroppers are defined as and 3. The dual-agent dual-delay deep deterministic policy gradient optimization method according to claim 2, characterized in that: The reward function includes penalties for violations of system constraints when interacting with the environment, where the reward function is set as: r n =tanh(SEE-λ1μ1-λ2μ2-λ3μ3) Among them, μ1, μ2 and μ3 are the penalties corresponding to the three constraints in the optimization objective; The weights used to guide the agent to achieve a preset convergence.

4. The dual-agent dual-delay deep deterministic policy gradient optimization method according to claim 1, characterized in that: The action space a of the first agent n,1 The first beamforming matrix W B and the second reflection matrix Ψ, the action space a of the second agent n,2 The third beamforming matrix W J The first reward function of the first agent and the second reward function of the second agent are set to be the same reward function; And / or, the channels are all Rician fading channel models, and the fading of the channels is calculated as follows: Among them, PL d is the path loss corresponding to the communication distance d, let PL d =-PL0-10α i,j log 10 (d / D0), d is the communication distance, D0 is the reference distance and its value is 1 meter, and the path loss PL0 corresponding to the reference distance is taken as 30dB; α i,j is the path loss factor, where i∈{B,J,I} and j∈{E,U k }; G LoS and G NLoS represent the line-of-sight LoS path and the Rayleigh fading component respectively; β is the Rayleigh attenuation factor.

5. The dual-agent dual-delay deep deterministic policy gradient optimization method according to any one of claims 1 to 4, characterized in that: Four fully connected hidden layers and four fully connected hidden layers are deployed in the Actor network and Critic network of the first and second agents; and / or, the number of neurons in the four fully connected hidden layers of the first agent is [800, 600, 512, 256], and the number of neurons in the four fully connected hidden layers of the second agent is [400, 300, 256, 128]; And / or, the learning rates of the Actor network and the Critic network of the first agent and the second agent are set to 0.0001 and 0.001 respectively; And / or, the first agent and the second agent are trained for 300 rounds, with 100 steps in each round.

6. A wireless communication secure transmission method, characterized in that: Applied to the communication process between a legitimate user and a base station, the wireless communication security transmission method includes: The base station uses a first beamforming matrix W B The optimization strategy of transmitting communication signals into space; The cooperative interferer is based on the third beamforming matrix W J The optimization strategy of transmitting interference signals to the eavesdropper to reduce the eavesdropper's eavesdropping rate; And, through the intelligent reconfigurable surface, according to the optimization strategy of the second reflection matrix Ψ, the interference signal is reflected to the eavesdropper to reduce the eavesdropping rate, and / or the useful signal is reflected to the legitimate user to assist the communication between the legitimate user and the base station, so as to improve the signal-to-interference-noise ratio of the legitimate user; Among them, the first beamforming matrix W B , the second reflection matrix Ψ, the third beamforming matrix W J The optimization strategy adopts the optimization strategy obtained by the dual-agent dual-delay deep deterministic policy gradient optimization method as described in any one of claims 1-5.

7. A dual-agent dual-delay deep deterministic policy gradient optimization system, characterized in that: include: A first agent and a second agent; The first agent is used to continuously interact with the environment and learn the first beamforming matrix W of the base station. B and the optimization strategy of the second reflection matrix Ψ of the intelligent reconfigurable surface; wherein the comprehensive channel gain H C and the comprehensive channel damage D C As the first state space s n,1 For the first beamforming matrix W B And the second reflection matrix ψ is optimized and learned; the comprehensive channel gain H C It includes the comprehensive channel state information from the base station to the eavesdropper, the intelligent reconfigurable surface, and the legitimate user, as well as the comprehensive channel state information from the intelligent reconfigurable surface to the legitimate user and the eavesdropper; the comprehensive channel damage D C including comprehensive channel state information from cooperative jammers to smart reconfigurable surfaces, legitimate users, and eavesdroppers; The second agent is used to continuously interact with the environment and learn the third beamforming matrix W of the cooperative interferer. J The optimization strategy of C As the second state space s n,2 The third beamforming matrix W J Optimize learning.

8. A wireless communication security transmission system, characterized in that: include: at least one smart reconfigurable surface, at least one base station, at least one cooperative jammer, at least one eavesdropper, and at least one legitimate user; The intelligent reconfigurable surface is equipped with multiple reflective surfaces; the intelligent reconfigurable surface, base station, and cooperative jammer are all equipped with multiple antennas, while the eavesdropper and legitimate user are equipped with a single antenna; The base station is configured to transmit the signal according to the first beamforming matrix W B The communication signal is transmitted to the space according to the optimization strategy of the third beamforming matrix W; the eavesdropper is used to eavesdrop when the legitimate user communicates with the base station; the cooperative interferer is used to J The intelligent reconfigurable surface is used to reflect the interference signal to the eavesdropper to reduce the eavesdropping rate of the eavesdropper according to the optimization strategy of the second reflection matrix Ψ during the communication between the legitimate user and the base station, and / or reflect the useful signal to the legitimate user to assist the communication between the legitimate user and the base station, so as to improve the signal-to-interference-noise ratio of the legitimate user; Among them, the first beamforming matrix W B , the second reflection matrix Ψ, the third beamforming matrix W J The optimization strategy adopts the optimization strategy obtained by the dual-agent dual-delay deep deterministic policy gradient optimization method as described in any one of claims 1-5.

9. A computing device, comprising: at least one processor, memory, and input-output unit; Wherein, the memory is used to store computer programs, and the processor is used to call the computer programs stored in the memory to execute: the steps of the dual-agent dual-delay deep deterministic policy gradient optimization method as described in any one of claims 1-5; or the steps of the wireless communication secure transmission method as described in claim 6.

10. A computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to execute: the steps of the dual-agent dual-delay deep deterministic policy gradient optimization method as described in any one of claims 1 to 5; or the steps of the wireless communication secure transmission method as described in claim 6.

Images