Wireless communication intelligent reflecting surface configuration method based on deep reinforcement learning

Abstract

The invention belongs to the field of wireless communication, and relates to a wireless communication intelligent reflecting surface configuration method based on deep reinforcement learning, which comprises the following steps of: constructing a system model which comprises an intelligent reflecting surface IRS, a base station BS and associated mobile user UE; constructing an optimization problem by taking maximization of the total transmission rate of the UE as a target according to the system model; the DDQN is improved by using an belonging-Greedy strategy, and an improved DDQN is obtained; converting an optimization problem into a Markov process, and solving the optimization problem by adopting an improved DDQN based on the Markov process to obtain a reflection mode of the intelligent reflection surface; according to the method, the belonging-Greedy strategy is applied to the DDQN, the threshold function is used in strategy loss to control the change between the new strategy and the old strategy according to the parameter belonging to the belongs, and the dominant function normalization process is further controlled, so that the DDQN training instability or performance reduction is prevented, and the DDQN is prevented from falling into the suboptimal strategy.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communications and relates to a method for configuring an intelligent reflective surface for wireless communications based on deep reinforcement learning. Background Art

[0002] As a passive device, the Reconfigurable Intelligent Surface (RIS) can intelligently reconstruct the wireless propagation environment in real time to enhance the signal strength by integrating a large number of low-cost passive reflective elements on the plane, thereby providing more degrees of freedom for the wireless link. At the same time, because it is a passive device, it greatly reduces the hardware cost and energy consumption.

[0003] The paper (Wang P, Fang J, Duan H, et al. Compressed channel estimation for intelligent reflecting surface-assisted millimeter wave systems [J]. IEEE signal processing letters, 2020, 27: 905-909) considers the channel estimation of intelligent reflecting surface (IRS)-assisted millimeter wave (mmWave) system, in which an IRS is deployed to assist data transmission from base station (BS) to users. The simulation results show that in order to jointly perform active beamforming and passive beamforming, it is necessary to obtain a large-size cascade channel matrix information. In order to reduce the training overhead, the inherent sparsity of the mmWave channel is utilized. By using the properties of Katri-Rao and Kronecker products, the sparse representation of the cascade channel is realized, and the cascade channel estimation is converted into a sparse signal recovery problem.

[0004] However, in the intelligent reflecting surface (IRS)-assisted millimeter wave (mmWave) system, although this method combines active beamforming and passive beamforming and can provide accurate channel estimation, the overall channel estimation overhead is large and is easily affected by the dynamics of the environment. Applying deep reinforcement learning to the model-free control of the intelligent reflecting surface can make it less dependent on channel state information and eliminate the need for a large amount of interaction between the intelligent reflecting surface and the wireless communication system, thereby reducing overhead and improving adaptability to different channel dynamics.

[0005] However, applying deep reinforcement learning to model-free control of intelligent reflective surfaces will result in long training time, unstable training, performance degradation, and falling into suboptimal strategies during the training process. Summary of the invention

[0006] In order to solve the above existing technical problems, the present invention adopts a wireless communication intelligent reflection surface configuration method based on deep reinforcement learning, comprising:

[0007] S1. Constructing a system model, wherein the system model includes: an intelligent reflecting surface IRS, a base station BS and its associated mobile user UE;

[0008] S2. Construct an optimization problem based on the system model to maximize the total transmission rate of the UE;

[0009] S3. Using the ∈-Greedy strategy to improve the DDQN, an improved DDQN is obtained; the ∈-Greedy strategy is the ∈-greedy strategy; the DDQN is a dual deep Q network;

[0010] S4. Convert the optimization problem into a Markov decision process, and use an improved DDQN based on the Markov decision process to solve the optimization problem and obtain the reflection pattern of the intelligent reflection surface.

[0011] The optimization problem is:

[0012]

[0013] Among them, P m is the objective function, i.e., the total transmission rate of mobile users, K is the number of mobile users, SINR k is the signal-to-noise ratio of mobile user k, θ[n] is the reflection pattern of the IRS reflector, n is the index of the IRS reflector, To quantize the phase, is a finite set of quantized phases, N R is the number of reflectors of the IRS, and j is an imaginary unit.

[0014] Signal-to-noise ratio (SINR) of mobile user k k for:

[0015]

[0016] Among them, h k is the channel model between the base station BS and the mobile user k, l is the index of the mobile user, m k 、m l are the precoding vectors of mobile user k and mobile user l respectively, is the additive white Gaussian noise between the base station BS and the mobile user k, and the superscript H represents the conjugate operation.

[0017] Converting the optimization problem into a Markov decision process involves:

[0018] Construct the state space S = {H *,θ}, construct the action space A={Δθ}, and construct the reward function R; where H * is the channel model between the BS and the mobile user UE, θ is the reflection pattern of the smart reflecting surface IRS, and Δθ is the incremental phase shift of the smart reflecting surface IRS.

[0019] The reward function R is:

[0020]

[0021] Among them, P th is the performance threshold.

[0022] Action space A = {Δθ} = {v(Ψ R (z))}; where v(Ψ R (z)) is the steering vector of IRS, Ψ R (z) is the direction cosine of IRS, {Ψ R (z)} is the set of direction cosines of IRS; {Ψ R (z)} includes multiple pairs of direction cosines with opposite directions and the same magnitude.

[0023] The improved DDQN includes: a Q network and a target Q network, wherein both the Q network and the target Q network include a GRU composite unit module; the improved DDQN is used to solve the optimization problem based on the Markov decision process, including:

[0024] Get the status s of the previous moment k ∈S and the current state s∈S, the state s at the previous moment k and the current state s are input into the Q network to obtain action a∈A; execute action a to obtain the state s′∈S and reward R at the next moment, and convert the five-tuple (s k ,s,a,R,s′) are stored in the experience replay buffer D; a batch of training samples are extracted from the experience replay buffer D, an improved DDQN is trained according to the training samples, a trained improved DDQN is obtained, an optimal incremental phase shift is obtained according to the trained improved DDQN, and a reflection pattern of the intelligent reflector IRS is obtained according to the optimal incremental phase shift;

[0025] The Q network includes: a first GRU compound unit module, a second GRU compound unit module and a fully connected layer; the Q network is for the state s at the previous moment k Processing with the current state s includes:

[0026] The state s at the previous moment k The channel model of the current state s is input into the first GRU composite unit module, and the state s of the previous moment is k The reflection mode of the intelligent reflection surface IRS in the current state s is input into the second GRU composite unit module;

[0027] The outputs of the first GRU compound unit module and the second GRU compound unit module are processed respectively by using a fully connected layer, the processed results are spliced, and the spliced ​​results are processed by using a fully connected layer to obtain an action.

[0028] The GRU composite unit module includes: a first state normalization layer, a second state normalization layer, a first linear layer, a second linear layer, a third linear layer, a fourth linear layer, a fifth linear layer, a first GRU gated recurrent unit, a second GRU gated recurrent unit and a Softmax activation function; the first GRU composite unit module is used to calculate the state s of the previous moment. k The channel model of the current state s is processed by:

[0029] Step 1: Input the channel model of the current state s into the first state normalization layer, and transform the state s of the previous moment into k The channel model input second state normalization layer;

[0030] Step 2: Input the output of the first state normalization layer to the first linear layer, and input the output of the second state normalization layer to the second linear layer;

[0031] Step 3: Input the output of the first linear layer into the first GRU gated recurrent unit, and input the output of the second linear layer into the second GRU gated recurrent unit.

[0032] Step 4: Input the output of the first GRU gated recurrent unit into the third linear layer, and input the output of the second GRU gated recurrent unit into the fourth linear layer.

[0033] Step 5: Concatenate the outputs of the third linear layer and the fourth linear layer, input the concatenated result into the fifth linear layer, input the output of the fifth linear layer into the activation function Softmax, and obtain the output result of the GRU compound unit module.

[0034] Improved DDQN’s total loss function L′ actor for:

[0035]

[0036] Among them, L actor is the strategy loss function, δ is the entropy coefficient, is the policy entropy,

[0037] The policy loss function is:

[0038]

[0039] in, is the current strategy of applying the ∈-Greedy strategy, is the strategy of the last moment when the ∈-Greedy strategy is applied, is the normalized advantage function of the strategy at the previous moment, is the advantage function, W(s,a) represents the value of taking action a in state s, V(s) represents the average value of following the strategy in state s, μ is the mean of the advantage function, σ is the standard deviation of the advantage function, ∈ is the probability of selecting the ∈-Greedy strategy, and g() is the threshold function.

[0040] The current strategy for applying the ∈-Greedy strategy is:

[0041]

[0042] in, is the current Q-network-based strategy, wp1-∈ indicates occurrence with probability 1-∈, P(a) is the probability of taking action a, and |A| is the size of the action space.

[0043] Beneficial effects:

[0044] 1. The present invention uses GRU to improve DDQN, so that the convergence and training stability of DDQN are stronger; 2. The present invention obtains the steering vector of IRS by performing discrete Fourier transform on the direction cosines of IRS, and constructs the action space using the steering vector of IRS. The set of direction cosines of IRS includes multiple pairs of direction cosines with opposite directions but the same magnitude. A pair of direction cosines with opposite directions but the same magnitude can enable the intelligent agent to quickly correct negative actions. Larger direction cosines can accelerate the transition of reflection mode and reduce training time; 3. The present invention applies ∈-Greedy strategy in DDQN, and in the strategy loss L actor A threshold function is used in the algorithm to control the change between the new strategy and the old strategy according to the parameter ∈ of the ∈-Greedy strategy, and the advantage function normalization process is further controlled to prevent the strategy update from being too aggressive, resulting in unstable training or performance degradation. Therefore, when maximizing the reward of the current action, it not only prevents DDQN training from being unstable or performance degradation, but also avoids DDQN from falling into a suboptimal strategy, thereby improving the exploration ability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning provided by an embodiment of the present invention;

[0046] Figure 2 A structural diagram of an IRS system model implemented by deep reinforcement learning provided in an embodiment of the present invention;

[0047] Figure 3 A diagram of a Q network structure provided by an embodiment of the present invention;

[0048] Figure 4 A structural diagram of a GRU composite unit module provided in an embodiment of the present invention;

[0049] Figure 5 A sequence diagram of a model-free IRS configuration provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] like Figure 1 , Figure 5 As shown, the present invention adopts a wireless communication intelligent reflection surface configuration method based on deep reinforcement learning, including:

[0052] S1. Constructing a system model, wherein the system model includes: an intelligent reflecting surface IRS, a base station BS and its associated mobile user UE;

[0053] like Figure 2 As shown, the base station is equipped with a linear antenna array, placed along the x-axis [1,0,0], with N antennas. B ; IRS is a linear array, placed along the y-axis [0,1,0], with N antennas R , controlled by an intelligent reflector controller; the mobile device UE is equipped with a single antenna, the number of mobile user UEs associated with each base station is K, and the number of base station antennas corresponds to the number of mobile device UEs. The unit antennas or reflectors of the base station and IRS are all half-wavelength spaced.

[0054] Without loss of generality, a typical scenario is used in the wireless communication of the system model of the present invention, that is, Time Division Duplexing (TDD) multi-user Multiple-Input Multiple-Output (MIMO). In TDD, by utilizing channel reciprocity, the BS can estimate the downlink channel from the pilot points of the uplink channel.

[0055] S2. Construct an optimization problem based on the system model to maximize the total transmission rate of the UE;

[0056] The deterministic Line of Sight (LOS) channel between the base station and the IRS is:

[0057]

[0058] The superscript H represents the conjugate operation, and the steering vector v R 、v B It is expressed as:

[0059]

[0060] Among them, N R,y is the number of IRS reflection units in the y-axis direction, N B,x is the number of antennas of the base station in the x-axis direction, j is the imaginary unit, and the direction cosine Ψ R ,Ψ B for:

[0061] Ψ R =[0,1,0]e BR =e BR (2)

[0062] Ψ B =[1,0,0]e BR =e BR (1)

[0063] Among them, the direction vector e BR Determined by the relative position of the base station and the user, that is:

[0064]

[0065] Among them, p B is the location of the base station, p R For the location of the IRS.

[0066] The fast fading non-line-of-sight (NLOS) component is Gaussian distributed, that is, represents a Gaussian distribution, represents the propagation distance, and k represents the fading amplitude.

[0067] The channel response vector between the base station BS and the mobile user k of the present invention is Channel response matrix between base station and IRS and the channel response vector between the intelligent reflector IRS and mobile user k is the Rician channel model; where H BR For example, it is represented as:

[0068]

[0069] in, is the ratio of the power of the LoS path to the power of the NLoS path.

[0070] Channel Response Vector and the channel response vector The generation method and channel response matrix H BR same.

[0071] Therefore, for IRS-assisted wireless communication, the channel model between the BS and a certain user k can be expressed as:

[0072]

[0073] Where Θ = diag{θ}, is the phase shift vector of the IRS. Therefore, the multi-user channel H * for:

[0074] H * =[h1,h2,…,h K ]

[0075] TDD multi-user MIMO consists of two phases, uplink pilot transmission and downlink data transmission:

[0076] In the uplink phase, pilot signals are transmitted simultaneously from multiple users, and the BS receives the pilot signal Y U It is expressed as:

[0077]

[0078] in, is the pilot mode, is additive white Gaussian noise, and H is the channel matrix. After receiving the pilot signal, the base station performs minimum mean square error (MMSE) estimation on the channel matrix and obtains the estimated channel matrix Right now:

[0079]

[0080] Where I is the identity matrix, is the noise variance at the base station side;

[0081] When S is a unitary matrix, the above formula is further expressed as:

[0082]

[0083] In the downlink phase, zero-forcing (ZF) precoding is used for data transmission, and the precoding matrix M is expressed as:

[0084]

[0085] Among them, m k is the precoding vector of user k, Used for power normalization; the signal y received by user k D,k Given by:

[0086]

[0087] Among them, x k Is a user and signal, is additive white Gaussian noise, is the noise variance on the mobile user side, is a Gaussian distribution; therefore, the signal-to-noise ratio SINR of user k k for:

[0088]

[0089] For a communication system, the performance index may be SINR, data rate, frame error rate (FER), etc. Without loss of generality, the present invention adopts the total transmission rate P of the mobile user m As performance indicators, namely:

[0090]

[0091] Among them, r k is the transmission rate of the kth user.

[0092] The optimization problem constructed is:

[0093]

[0094] Among them, P m is the performance metric of the wireless system to be optimized, From a finite set There are 2 selected r +1 possible value of quantization phase, r is a selectable parameter.

[0095] S3. Using the ∈-Greedy strategy to improve DDQN, an improved DDQN is obtained; the ∈-Greedy strategy is the ∈-greedy strategy; the DDQN is a dual deep Q network;

[0096] S4. Convert the optimization problem into a Markov decision process, and use the improved DDQN based on the Markov decision process to solve the optimization problem and obtain the intelligent reflector configuration result.

[0097] Converting the optimization problem into a Markov decision process involves:

[0098] The IRS controller is regarded as an intelligent agent that can autonomously interact with the environment through IRS to meet the designed goals.

[0099] Construct the state space S = {H * ,θ}, construct the action space A={Δθ}, and construct the reward function R; where H * is the channel model between the BS and the mobile user UE, θ is the reflection pattern of the intelligent reflection surface IRS, Δθ is the incremental phase shift, which can be expressed as:

[0100] θ′=θ⊙Δθ

[0101] Where ⊙ is the Hadamard product, θ′ is the reflection pattern with the incremental phase shift added, θ is the original reflection pattern, and Δθ is the incremental phase shift of θ.

[0102] R is defined as: Among them, P th is the performance threshold, when P m Less than P th When , a penalty of -100 is added to incentivize IRS to maximize performance while maintaining acceptable performance above the threshold.

[0103] Preferably, according to simulation experiments, P th Set to 10.

[0104] Action space A = {Δθ} = {v(Ψ R (z))}; where v(Ψ R (z)) is the steering vector of IRS, Ψ R (z) is the direction cosine of IRS, {Ψ R (z)} is the set of direction cosines of IRS; {Ψ R (z)} includes multiple pairs of direction cosines with opposite directions and the same magnitude. For example, when the action space size is 5, set the action set Among them, v(Ψ R ) is the steering vector (without loss of generality, assuming that the reflector array of the IRS is a uniform linear array (ULA)), that is:

[0105]

[0106] When Δθ=v(0), the substate θ remains unchanged; and in the opposite direction, which enables the agent to quickly correct negative actions; and Used to accelerate the transition of reflection mode.

[0107] The improved DDQN based on Markov decision process is used to solve the optimization problems including:

[0108] The improved DDQN includes: a Q network and a target Q network, wherein both the Q network and the target Q network include a GRU compound unit module, a fully connected layer, and a series layer; the optimization problem solved by using the improved DDQN based on the Markov process includes:

[0109] Get the status s of the previous moment k ∈S and the current state s∈S, the state s at the previous moment k and the current state s are input into the Q network to obtain action a∈A; execute action a to obtain the state s′∈S and reward R at the next moment, and convert the five-tuple (s k ,s,a,R,s′) are stored in the experience replay buffer D; a batch of training samples are extracted from the experience replay buffer D, an improved DDQN is trained according to the training samples, a trained improved DDQN is obtained, an optimal incremental phase shift is obtained according to the trained improved DDQN, and a reflection pattern of the intelligent reflector IRS is obtained according to the optimal incremental phase shift;

[0110] like Figure 3 As shown, the Q network is in the state s of the previous moment k Processing with the current state s includes:

[0111] The state s at the previous moment k The channel model of the current state s is input into the first GRU composite unit module, and the state s of the previous moment is k The reflection mode of the intelligent reflection surface IRS in the current state s is input into the second GRU composite unit module;

[0112] The output of the first GRU composite unit module is input into the three-layer fully connected layer F i H , the first fully connected layer F1 H The output of the third fully connected layer F3 H Add together to get the characteristics of the channel model;

[0113] The output of the second GRU composite unit module is input into the three-layer fully connected layer F i θ , the first fully connected layer F1 θ The output of the third fully connected layer F3 θ Add together to obtain the characteristics of the reflection pattern of IRS;

[0114] The features of the channel model and the features of the reflection pattern of the IRS are input into the series layer for splicing, and the splicing results are input into two fully connected layers to obtain the action.

[0115] The target Q network has the same structure as the Q network.

[0116] In one embodiment, Figure 4 As shown, the GRU composite unit module includes: a first state normalization layer (StateNormalization), a second state normalization layer, a first linear layer (Linear), a second linear layer, a third linear layer, a fourth linear layer, a fifth linear layer, a first GRU gated recurrent unit, a second GRU gated recurrent unit and a Softmax activation function.

[0117] The first GRU compound unit module is the state s of the previous moment k The channel model of the current state s is processed by:

[0118] Step 1: Input the channel model of the current state s into the first state normalization layer, and transform the state s of the previous moment into k The channel model input second state normalization layer;

[0119] Step 2: Input the output of the first state normalization layer to the first linear layer, and input the output of the second state normalization layer to the second linear layer;

[0120] Step 3: Input the output of the first linear layer into the first GRU gated recurrent unit, and input the output of the second linear layer into the second GRU gated recurrent unit.

[0121] Step 4: Input the output of the first GRU gated recurrent unit into the third linear layer, and input the output of the second GRU gated recurrent unit into the fourth linear layer.

[0122] Step 5: Concatenate the outputs of the third linear layer and the fourth linear layer, input the concatenated result into the fifth linear layer, input the output of the fifth linear layer into the activation function Softmax, and obtain the output result of the GRU compound unit module.

[0123] Improved DDQN’s total loss function L′ actor for:

[0124]

[0125] Among them, L actor is the policy loss function of the Q network, δ is the entropy coefficient, is the policy entropy, The greater the entropy of a policy, the more evenly the probability of choosing each action is distributed.

[0126] Preferably, the value of δ is 0.01.

[0127] L actor The loss function is given by:

[0128]

[0129] in, is the current strategy of applying the ∈-Greedy strategy, is the strategy of the last moment when the ∈-Greedy strategy is applied, is the normalized advantage function of the strategy at the previous moment, is the advantage function, Q(s,a) represents the value of taking action a in state s, V(s) represents the average value of following the strategy in state s, μ is the mean of the advantage function, σ is the standard deviation of the advantage function, ∈ is the probability of selecting the ∈-Greedy strategy, and g() is the threshold function.

[0130] Apply the ∈-Greedy strategy in DDQN and set the policy loss L actor A threshold function is used in the algorithm to control the change between the new strategy and the old strategy according to the parameter ∈ of the ∈-Greedy strategy, and further control the normalization process to prevent the strategy update from being too radical, resulting in unstable training or performance degradation. This prevents DDQN from falling into a suboptimal strategy while preventing the strategy update from being too radical, thereby improving the algorithm's exploration ability.

[0131] The current strategy for applying the ∈-Greedy strategy is:

[0132]

[0133] in, is the current Q-network-based strategy, wp1-∈ indicates occurrence with probability 1-∈, P(a) is the probability of taking action a, and |A| is the size of the action space.

[0134] Preferably, ∈ is initially set to 1 and decreases exponentially at a rate of 0.5 until it reaches a lower limit of 0.2.

[0135] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning, characterized in that: include: S1. Constructing a system model, wherein the system model includes: an intelligent reflecting surface IRS, a base station BS and its associated mobile user UE; S2. Construct an optimization problem based on the system model to maximize the total transmission rate of the UE; S3. Using the ∈-Greedy strategy to improve the DDQN, an improved DDQN is obtained; the ∈-Greedy strategy is the ∈-greedy strategy; the DDQN is a dual deep Q network; S4. Convert the optimization problem into a Markov decision process, and use an improved DDQN based on the Markov decision process to solve the optimization problem and obtain the reflection pattern of the intelligent reflection surface.

2. According to claim 1, a method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning is characterized in that: The optimization problem is: Among them, P m is the objective function, i.e., the total transmission rate of mobile users, K is the number of mobile users, SINR k is the signal-to-noise ratio of mobile user k, θ[n] is the reflection pattern of the IRS reflector, n is the index of the IRS reflector, To quantize the phase, is a finite set of quantized phases, N R is the number of reflectors of the IRS, and j is an imaginary unit.

3. According to claim 2, a method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning is characterized in that: Signal-to-noise ratio (SINR) of mobile user k k for: Among them, h k is the channel model between the base station BS and the mobile user k, l is the index of the mobile user, m k 、m l are the precoding vectors of mobile user k and mobile user l respectively, is the additive white Gaussian noise between the base station BS and the mobile user k, and the superscript H represents the conjugate operation.

4. According to claim 2, a method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning is characterized in that: Converting the optimization problem into a Markov decision process involves: Construct the state space S = {H*, θ}, construct the action space A = {Δθ}, and construct the reward function R; where H* is the channel model between the BS and the mobile user UE, θ is the reflection pattern of the smart reflecting surface IRS, and Δθ is the incremental phase shift of the smart reflecting surface IRS.

5. According to claim 4, a method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning is characterized in that: The reward function R is: Among them, P th is the performance threshold.

6. The method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning according to claim 4, characterized in that: Action space A = {Δθ} = {v(Ψ R (z))}; Among them, v(Ψ R (z)) is the steering vector of IRS, Ψ R (z) is the direction cosine of IRS, {Ψ R (z)} is the set of direction cosines of IRS, {Ψ R (z)} includes multiple pairs of direction cosines with opposite directions and the same magnitude.

7. The method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning according to claim 4, characterized in that: The improved DDQN includes: a Q network and a target Q network, wherein both the Q network and the target Q network include a GRU composite unit module; the improved DDQN is used to solve the optimization problem based on the Markov decision process, including: Get the status s of the previous moment k ∈S and the current state s∈S, the state s at the previous moment k and the current state s are input into the Q network to obtain action a∈A; execute action a to obtain the state s′∈S and reward R at the next moment, and convert the five-tuple (s k , s, a, R, s′) are stored in the experience replay buffer D; a batch of training samples are extracted from the experience replay buffer D, an improved DDQN is trained according to the training samples, a trained improved DDQN is obtained, an optimal incremental phase shift is obtained according to the trained improved DDQN, and a reflection pattern of the intelligent reflector IRS is obtained according to the optimal incremental phase shift; The Q network includes: a first GRU compound unit module, a second GRU compound unit module and a fully connected layer; the Q network is for the state s at the previous moment k Processing with the current state s includes: The state s at the previous moment k The channel model of the current state s is input into the first GRU composite unit module, and the state s of the previous moment is k The reflection mode of the intelligent reflection surface IRS in the current state s is input into the second GRU composite unit module; The outputs of the first GRU compound unit module and the second GRU compound unit module are processed respectively by using a fully connected layer, the processed results are spliced, and the spliced ​​results are processed by using a fully connected layer to obtain an action.

8. The method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning according to claim 7, characterized in that: The GRU composite unit module includes: a first state normalization layer, a second state normalization layer, a first linear layer, a second linear layer, a third linear layer, a fourth linear layer, a fifth linear layer, a first GRU gated recurrent unit, a second GRU gated recurrent unit and a Softmax activation function; the first GRU composite unit module is used to calculate the state s at the previous moment. k The channel model of the current state s is processed by: Step 1: Input the channel model of the current state s into the first state normalization layer, and transform the state s of the previous moment into k The channel model input second state normalization layer; Step 2: Input the output of the first state normalization layer to the first linear layer, and input the output of the second state normalization layer to the second linear layer; Step 3: Input the output of the first linear layer into the first GRU gated recurrent unit, and input the output of the second linear layer into the second GRU gated recurrent unit. Step 4: Input the output of the first GRU gated recurrent unit into the third linear layer, and input the output of the second GRU gated recurrent unit into the fourth linear layer. Step 5: Concatenate the outputs of the third linear layer and the fourth linear layer, input the concatenated result into the fifth linear layer, input the output of the fifth linear layer into the activation function Softmax, and obtain the output result of the GRU compound unit module.

9. The method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning according to claim 7, characterized in that: Improved DDQN’s total loss function L′ actor for: Among them, L actor is the strategy loss function, δ is the entropy coefficient, is the policy entropy, is the current strategy of applying the ∈-Greedy strategy, is the strategy of the last moment when the ∈-Greedy strategy is applied, is the normalized advantage function of the strategy at the previous moment, is the advantage function, Q(s,a) represents the value of taking action a in state s, V(s) represents the average value of following the strategy in state s, μ is the mean of the advantage function, σ is the standard deviation of the advantage function, ∈ is the probability of selecting the ∈-Greedy strategy, and g() is the threshold function.

10. A method for configuring a wireless communication intelligent reflector surface based on deep reinforcement learning according to claim 9, characterized in that: The current strategy for applying the ∈-Greedy strategy is: in, is the current Q-network-based strategy, wp1-∈ indicates occurrence with probability 1-∈, P(a) is the probability of taking action a, and |A| is the size of the action space.

Images