A resource allocation method for IRS-assisted SWIPT-D2D system based on deep reinforcement learning

Abstract

The present invention relates to the field of wireless communication technology, and in particular to an IRS-assisted SWIPT-D2D system resource allocation method based on deep reinforcement learning. The steps are as follows: first, the received signals of the base station and each receiving user (D2D Receiver, DR) are obtained according to the channel gain of the system; secondly, the reflection coefficient on the IRS and the transmission power and time switching factor of each transmitting user (D2D Transmitter, DT) are obtained, and the information decoding rate and collected energy of the DR are obtained; under the information transmission rate constraint conditions of the cellular user and the D2D user are met, a maximum system energy collection optimization model is constructed; finally, the DQN-DDPG algorithm based on deep reinforcement learning is used to jointly optimize the time switching factor, IRS phase shift and D2D transmission power to maximize the total energy collected by the system. The DQN-DDPG algorithm based on deep reinforcement learning proposed in the present invention uses a DQN network to solve the D2D transmission power and IRS phase shift, and a DDPG network to solve the time switching factor, converting the redundant radio frequency signal into energy, thereby reducing the total energy consumption of the system.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technologies, and in particular to an IRS-assisted SWIPT-D2D system resource allocation method based on deep reinforcement learning. Background Art

[0002] SWIPT technology can achieve the simultaneous transmission of energy and information, which is of great significance for solving problems such as energy consumption, spectrum efficiency and interference control. In the field of IoT, SWIPT technology can effectively solve the energy supply problem of low-power and charging-restricted devices. In order to realize SWIPT technology, the RF signal needs to be divided into two parts, one for transmitting information between users and the other for transmitting energy. At present, signal splitting is mainly based on three aspects: time slot, power and antenna.

[0003] D2D communication technology enables D2D users to reuse the spectrum resources of cellular users, thus realizing information transmission between D2D users and improving the spectrum utilization of the system. However, D2D communication will also introduce new interference sources to the system, increasing the interference to cellular users. Therefore, it is important to reasonably allocate spectrum resources and control the transmission power to reduce system interference and improve system performance.

[0004] As a new revolutionary technology, intelligent reflecting surface (IRS) can achieve efficient use of spectrum and energy. IRS is composed of a large number of passive low-cost reflecting units, each of which can adjust the phase and amplitude of the incident electromagnetic wave and reflect it. Therefore, IRS can be used to design passive beams, that is, to enhance the desired signal and suppress interference by changing the reflection coefficient (including phase and amplitude) of each reflecting unit. The typical architecture of IRS includes an intelligent controller and a three-layer structure (reflection element, copper backplane and control circuit board). The controller connected to the IRS can intelligently adjust the reflection coefficient and communicate with other network components to achieve reconfiguration of the wireless propagation environment, thereby improving the anti-interference ability of the D2D communication system at the physical layer level.

[0005] Artificial intelligence has been a hot research topic in recent years, covering knowledge from multiple fields such as probability theory, statistics, and computers. As one of the most important technologies in artificial intelligence, machine learning focuses on how computers can acquire knowledge and skills through learning, thereby simulating human thinking and behavior. Machine learning has been maturely applied in the industrial sector, and has had a profound impact in the fields of data mining, pattern recognition, and recommendation systems. Summary of the invention

[0006] The main contributions of this invention can be divided into the following three aspects:

[0007] IRS-assisted SWIPT-D2D communication system

[0008] At present, most of the research on IRS mainly uses IRS for wireless information transmission, and does not consider more application scenarios. This paper designs an IRS-assisted SWIPT-D2D communication system, which combines IRS technology, SWIPT technology and D2D technology, which can not only improve spectrum utilization, but also realize system autonomous energy replenishment, making the model more practical.

[0009] Resource allocation scheme for time-slot switching energy harvesting

[0010] For the IRS-assisted SWIPT-D2D communication scenario with time-slot switching energy harvesting, D2D users can both transmit information and harvest energy, and switch between the two working modes through the time-slot switching factor. Firstly, under the constraints of QoS of cellular users and D2D users, the maximization energy harvesting optimization problem is constructed. Since the time switching factor is a continuous variable, the DQN-DDPG algorithm is proposed, in which DQN is used to solve the D2D user transmit power and IRS phase shift, and DDPG solves the time switching factor. Finally, the performance of the two algorithms is verified by simulation results.

[0011] The technical solution of the present invention is implemented as follows:

[0012] The steps of a resource allocation method for an IRS-assisted SWIPT-D2D system based on deep reinforcement learning are as follows:

[0013] Step 1: A communication system based on a multiplexing uplink spectrum resource in a device-to-device (D2D) communication assisted by an intelligent reflecting surface (IRS), including a base station (BS), a cellular user, and K pairs of D2D users, where each pair of D2D users includes a transmitting user (D2D Transmitter, DT) and a receiving user (D2D Receiver, DR);

[0014] Step 2: Calculate the received signals of the base station and each DR based on the channel gains from the cellular user to the BS, IRS, and DR, from the DT to the DR, IRS, and BS, and from the IRS to the BS and DR;

[0015] Step 3: Calculate the information decoding rate and collected energy of DR based on the reflection coefficient on IRS, the transmit power of each DT, and the time switching factor, where the time switching factor is ξ;

[0016] Step 4: Construct a maximum system energy harvesting optimization model based on the information transmission rate constraints of cellular users and D2D users;

[0017] Step 5: Use the DQN-DDPG algorithm based on deep reinforcement learning to jointly optimize the D2D transmission power, time switching factor and IRS phase shift to maximize the system energy harvesting.

[0018] The channel gain of the system specifically includes:

[0019] The channel gains from cellular users to BS, IRS, and DR are The channel gains from DT to DR, IRS, and BS are The channel gains from IRS to BS and DR are

[0020] The signal received at the base station is:

[0021]

[0022] The received signal of the i-th DR user is:

[0023]

[0024] Among them, P C , are the transmit powers of the cellular user and the i-th DT respectively; Θ is the IRS phase shift matrix, satisfying Main diagonal θ n =(0,2π) represents the phase shift of the nth (1≤n≤N) reflection element; x is the transmitted signal of the cellular user, s i is the transmitted signal of the i-th DT, E(x 2 )=E(s i 2 )=1; n1 and n2 represent additive Gaussian white noise, satisfying n i ~CN(0,σ 2 ),i=1,2.

[0025] The information decoding rate of the DR is:

[0026]

[0027] The system collects energy as:

[0028]

[0029] Among them, ξ i is the time switching factor of the i-th DR, and η is the energy conversion efficiency.

[0030] The optimization model for maximizing system energy harvesting is:

[0031]

[0032] stC1:γ B ≥γ min

[0033]

[0034] Where ξ=[ξ1,ξ2,...,ξ K ], C1 is the SINR constraint for cellular users, γ min is the minimum SINR; C2 is the D2D user transmission rate constraint, R min represents the minimum transmission rate at which D2D users can communicate normally; C3 describes the energy collection constraint of D2D users, E min is recorded as the minimum energy collection limit for D2D users; C4 represents the IRS constant modulus constraint; C5 is the DT transmission power constraint, P max is the maximum transmit power of DT.

[0035] The DQN-DDPG algorithm jointly optimizes D2D transmit power, time switching factor and IRS phase shift as follows:

[0036] A DQN-DDPG resource allocation network is constructed. The DDPG network is used to learn the optimal time switching factor, and the DQN network is used to learn D2D power allocation and IRS phase shift selection.

[0037] The algorithm consists of two parts: online exploration and offline training: (1) Online exploration phase: First, the system is initialized and the current state value s is continuously obtained. t , as the input of the DQN-DDPG network and outputs the joint action a t ={a t1 ,a t2 ,a t3}; Agent performs action a t Then get an instantaneous reward value r t and reaches the next state s t+1 , and at the same time {s t ,a t ,r t ,s t+1} is put into the experience pool as a set of data samples; (2) Offline training phase: According to the experience replay mechanism, batch data is randomly sampled from the experience pool, and the true target Q value is defined as:

[0038] y i =r i +γmaxQ(s i+1 ,a i+1 ; θ, α, β)

[0039] DQN-DDPG can split the Q-value function model into two parts. The first part is a function related only to the state, and the second part is a function related to both the state and the action. Therefore, the value function can be expressed as:

[0040] Q(s i+1 ,a i+1 ;θ,α,β)=V(s i+1 ;θ,α)+A(s i+1 ,a i+1 ; θ, β)

[0041] The loss function of the DQN-DDPG network is:

[0042]

[0043] Like DQN, DQN-DDPG first fixes the parameters in the target network, and then copies the parameters in the current network to the target network after training the current network for a period of time. Different from the DQN network parameter update, the DDPG network adopts a soft update method, that is, the parameter value is updated a little bit each time, which is expressed as:

[0044] θ - ←τθ+(1-τ)θ -

[0045] Among them, τ is the soft update coefficient, which is generally taken as 0.1 or 0.01.

[0046] It can be seen from the above technical solutions that the present invention applies SWIPT technology and IRS technology to the D2D communication system, and proposes a resource allocation solution based on machine learning for optimization problems such as power control and IRS phase shift in D2D communication scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is a schematic diagram of the structure of the IRS-assisted SWIPT-D2D communication system;

[0049] Figure 2 It is the DQN-DDPG network model

[0050] Figure 3 This is the DQN-DDPG resource allocation algorithm flow chart

[0051] Figure 4 is the relationship between energy harvesting and the number of IRS reflective elements N;

[0052] Figure 5 is the relationship between energy harvesting and the number of D2D users K; DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] like Figure 1 As shown, the method described in this embodiment is applied to the uplink of the IRS-assisted SWIPT-D2D communication system. A base station (BS) is surrounded by a cellular user and K pairs of D2D users, where each pair of D2D users includes a transmitting user (D2D Transmitter, DT) and a receiving user (D2D Receiver, DR). In addition, an IRS composed of a large number of built-in programmable components is deployed in the cell to reflect directional beams to the target receiver. The IRS includes N reflection units, assuming that the phase shift of each reflection unit is discrete 4-bit, the phase shift range is [0,2π], and the amplitude of the IRS transmitting unit is 1.

[0055] In this embodiment, the specific steps are as follows:

[0056] Step 1: Set the channel gains from cellular users to BS, IRS, and DR respectively. The channel gains from DT to DR, IRS, and BS are The channel gains from IRS to BS and DR are

[0057] Step 2: According to step 1, the signal received at the base station is:

[0058]

[0059] The received signal of the i-th DR user is:

[0060]

[0061] Among them, P C , are the transmit powers of the cellular user and the i-th DT respectively; Θ is the IRS phase shift matrix, satisfying Main diagonal θ n =(0,2π) represents the phase shift of the nth (1≤n≤N) reflection element; x is the transmitted signal of the cellular user, s i is the transmitted signal of the i-th DT, E(x 2 )=E(s i 2 )=1; n1 and n2 represent additive Gaussian white noise, satisfying n i ~CN(0,σ 2 ),i=1,2.

[0062] Step 3: Calculate the information decoding rate and collected energy of DR based on the reflection coefficient on IRS, the transmit power of each DT, and the time switching factor;

[0063] In a time slot T, ξ i T is used for information transmission, (1-ξ i )T is used for energy harvesting, and the information decoding rate and energy harvesting of DR are:

[0064]

[0065] Among them, ξ i is the time switching factor of the i-th DR, and η is the energy conversion efficiency.

[0066] Step 4: Construct a maximum system energy harvesting optimization model based on the information transmission rate constraints of cellular users and D2D users;

[0067]

[0068] stC1:γ B ≥γ min

[0069]

[0070] Where ξ=[ξ1,ξ2,...,ξ K ], C1 is the SINR constraint for cellular users, γ min is the minimum SINR; C2 is the D2D user transmission rate constraint, R min represents the minimum transmission rate at which D2D users can communicate normally; C3 describes the energy collection constraint of D2D users, E min is recorded as the minimum energy collection limit for D2D users; C4 represents the IRS constant modulus constraint; C5 is the DT transmission power constraint, P max is the maximum transmit power of DT.

[0071] Step 5: Use the DQN-DDPG algorithm based on deep reinforcement learning to jointly optimize the D2D transmission power, time switching factor and IRS phase shift to maximize the system energy harvesting.

[0072] like Figure 2 As shown in Figure 1, a DQN-DDPG resource allocation network is constructed. The DDPG network is used to learn the optimal time switching factor, and the DQN network is used to learn D2D power allocation and IRS phase shift selection. The algorithm consists of two parts: online exploration and offline training: (1) Online exploration phase: First, the system is initialized and the current state value s is continuously obtained. t , as the input of the DQN-DDPG network and outputs the joint action a t ={a t1 ,a t2 ,a t3}. Agent performs action a t Then get an instantaneous reward value r t and reaches the next state s t+1 , and at the same time {s t ,a t ,r t ,s t+1} is put into the experience pool as a set of data samples. (2) Offline training phase: Figure 3 As shown, according to the experience replay mechanism, batch data is randomly sampled from the experience pool, and the true target Q value is defined as:

[0073] y i =r i +γmaxQ(s i+1 ,a i+1 ; θ, α, β)

[0074] DQN-DDPG can split the Q-value function model into two parts. The first part is a function related only to the state, and the second part is a function related to both the state and the action. Therefore, the value function can be expressed as:

[0075] Q(s i+1 ,a i+1 ;θ,α,β)=V(s i+1 ;θ,α)+A(s i+1 ,a i+1 ; θ, β)

[0076] The loss function of the DQN-DDPG network is:

[0077]

[0078] Like DQN, DQN-DDPG first fixes the parameters in the target network, and after the current network is trained for a period of time, the parameters in the current network are copied to the target network. Unlike the DQN network parameter update, the DDPG network adopts a soft update method, that is, the parameter value is updated a little bit each time, which is expressed as:

[0079] θ - ←τθ+(1-τ)θ -

[0080] Among them, τ is the soft update coefficient, which is generally taken as 0.1 or 0.01.

[0081] A comparison between the energy harvesting solution of the present invention and other existing solutions is given below to make the advantages and features of the present invention more obvious.

[0082] Figure 4 is the relationship between system energy harvesting and the number of IRS reflection elements N, where the number of D2D users K = 4 and the lower limit of D2D user transmission rate R min =1bps / Hz. As can be seen from the figure, the DQN-DDPG algorithm has the highest energy collection. Among them, the IRS-assisted SWIPT-D2D solution is significantly better than the solution without IRS, and the total energy collection of the system increases with the increase in the number of IRS reflection elements. This is because as the number of reflection elements increases, the spatial degrees of freedom and diversity gain obtained by IRS will become larger, thereby improving the system performance.

[0083] Figure 5 The relationship between the total energy harvesting of the system and the number of D2D users K, where the number of IRS transmitting elements N = 20 and the lower limit of the D2D user transmission rate R min =1bps / Hz. As can be seen from the figure, as the number of D2D users increases, the total energy collection of the system also increases. The reason is that as the number of D2D users increases, the number of devices collecting energy in the system will also increase, so the total energy collection value will also increase. And the comparative analysis shows that the energy collection efficiency of the proposed scheme is better than other benchmark schemes. In particular, when the number of D2D users K=4, the energy collected by the DQN-DDPG algorithm is 0.177mW higher than the parallel CNN algorithm, 0.712mW higher than the block coordinate descent method, 2.567mW higher than the random phase shift method, and 3.312mW higher than the scheme without IRS.

[0084] It can be seen that the resource allocation method for IRS-assisted SWIPT-D2D communication based on deep reinforcement learning proposed in this embodiment maximizes the total energy collection of D2D users under the constraints of ensuring normal communication between cellular users and D2D users, and can convert excess RF signals into energy, extend the service life of the equipment, and meet the needs of green communication.

[0085] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An IRS-assisted SWIPT-D2D system resource allocation method based on deep reinforcement learning, characterized in that: Here are the steps: Step 1: A communication system based on a multiplexing uplink spectrum resource in a device-to-device (D2D) communication assisted by an intelligent reflecting surface (IRS), including a base station (BS), a cellular user, and K pairs of D2D users, where each pair of D2D users includes a transmitting user (D2D Transmitter, DT) and a receiving user (D2D Receiver, DR); Step 2: Calculate the received signals of the base station and each DR based on the channel gains from the cellular user to the BS, IRS, and DR, from the DT to the DR, IRS, and BS, and from the IRS to the BS and DR; Step 3: Calculate the information decoding rate and collected energy of the DR according to the reflection coefficient on the IRS, the transmit power of each DT, and the time switching factor, wherein the information decoding rate and collected energy of the DR are: The information decoding rate of DR is: The energy collected is: Among them, ξ i is the time switching factor of the i-th DR, η is the energy conversion efficiency; Step 4: Construct a maximum system energy harvesting optimization model based on the information transmission rate constraints of cellular users and D2D users; Step 5: Use the DQN-DDPG algorithm based on deep reinforcement learning to jointly optimize the D2D transmission power, time switching factor and IRS phase shift to maximize the system energy harvesting.

2. According to claim 1, a deep reinforcement learning-based IRS-assisted SWIPT-D2D system resource allocation method is characterized in that: The channel gain of the system specifically includes: The channel gains from cellular users to BS, IRS, and DR are The channel gains from DT to DR, IRS, and BS are The channel gains from IRS to BS and DR are 3. According to claim 1, a method for resource allocation of an IRS-assisted SWIPT-D2D system based on deep reinforcement learning, characterized in that: The received signals of the base station and each DR are: The signal received at the base station is: The received signal of the i-th DR user is: Among them, P C , are the transmit powers of the cellular user and the i-th DT respectively; Θ is the IRS phase shift matrix, satisfying Main diagonal θ n =(0,2π) represents the phase shift of the nth (1≤n≤N) reflection element; x is the transmitted signal of the cellular user, s i is the transmitted signal of the i-th DT, E(x 2 )=E(s i 2 )=1; n1 and n2 represent additive Gaussian white noise, satisfying n i ~CN(0,σ 2 ),i=1,2.

4. According to claim 1, a method for resource allocation of an IRS-assisted SWIPT-D2D system based on deep reinforcement learning, characterized in that: The optimization model for maximizing system energy harvesting is: s.t.C1:γ B ≥γ min Where ξ=[ξ1,ξ2,...,ξ K ], C1 is the SINR constraint for cellular users, γ min is the minimum SINR; C2 is the D2D user transmission rate constraint, R min represents the minimum transmission rate at which D2D users can communicate normally; C3 describes the energy collection constraint of D2D users, E min is recorded as the minimum energy collection limit for D2D users; C4 represents the IRS constant modulus constraint; C5 is the DT transmission power constraint, P max is the maximum transmit power of DT.

5. According to claim 1, a method for resource allocation of IRS-assisted SWIPT-D2D system based on deep reinforcement learning, characterized in that: The DQN-DDPG algorithm jointly optimizes D2D transmit power, time switching factor and IRS phase shift as follows: A DQN-DDPG resource allocation network is constructed. The DDPG network is used to learn the optimal time switching factor, and the DQN network is used to learn D2D power allocation and IRS phase shift selection. The algorithm consists of two parts: online exploration and offline training: (1) Online exploration phase: First, the system is initialized and the current state value s is continuously obtained. t , as the input of the DQN-DDPG network and outputs the joint action a t ={a t1 ,a t2 ,a t3 }; Agent performs action a t Then get an instantaneous reward value r t and reaches the next state s t+1 , and at the same time {s t ,a t ,r t ,s t+1 } is put into the experience pool as a set of data samples; (2) Offline training phase: According to the experience replay mechanism, batch data is randomly sampled from the experience pool, and the true target Q value is defined as: y i =r i +γmaxQ(s i+1 ,a i+1 ;i,a,b) DQN-DDPG can split the Q-value function model into two parts. The first part is a function related only to the state, and the second part is a function related to both the state and the action. Therefore, the value function can be expressed as: Q(s i+1 ,a i+1 ;θ,α,β)=V(s i+1 ;θ,α)+A(s i+1 ,a i+1 ;i,b) The loss function of the DQN-DDPG network is: Like DQN, DQN-DDPG first fixes the parameters in the target network, and then copies the parameters in the current network to the target network after training the current network for a period of time. Different from the DQN network parameter update, the DDPG network adopts a soft update method, that is, the parameter value is updated a little bit each time, which is expressed as: i - ←τθ+(1-τ)θ - Among them, τ is the soft update coefficient, which is generally taken as 0.1 or 0.01.

Images