Dynamic reflection coefficient adjusting method for double intelligent reflecting surfaces based on deep reinforcement learning

Abstract

The invention discloses a method for adjusting dynamic reflection coefficients of double intelligent reflecting surfaces based on deep reinforcement learning. At present, the complexity and dynamic change of the environment are important factors influencing intelligent reflecting surface phase regulation and control, and long-term phase regulation and control are difficult in the dynamic and complex environment. According to the method, on the basis of deep reinforcement learning (DRL), continuous adjustment is carried out through feedback of environment information, an optimal strategy is achieved, and adjustment of the phase and the amplitude of the double-intelligent-reflection-surface unit is carried out. According to the method, a flexible, efficient and automatic method can be provided for a double-intelligent-reflection-surface wireless communication system to achieve reflection coefficient optimization control, especially in the face of constantly changing environments and complex system dynamic states. In this way, the energy efficiency ratio of the wireless network can be improved, the coverage range can be expanded, and the user experience can be improved. In addition, the convergence rate is improved by carrying out distributed training on the method provided by the invention. Simulation results show that the method of the invention is effective and feasible, and the method of the invention has a fast convergence rate.

Description

Technical Field

[0001] The present invention belongs to the fields of wireless communications, intelligent reflecting surfaces, deep reinforcement learning, and DPPO, and in particular relates to a dual intelligent reflecting surface collaborative transmission optimization method. Background Art

[0002] The emergence of smart metasurface (smart reflective surface) technology is a revolutionary supplement to modern wireless communication systems. It uses a series of controllable passive units to intelligently adjust the propagation characteristics of electromagnetic waves. The core advantage of smart reflective surfaces is that they can finely control the reflection direction, phase and amplitude of signals, which allows them to effectively bypass obstacles even in complex urban environments, providing alternative propagation paths for more reliable signal transmission.

[0003] At present, the research on smart reflective surfaces is all based on the assumption of prior knowledge of the known environment. However, with the development of communication technology, the Internet of Things has become a highly complex system, and this prior knowledge is difficult to obtain in actual situations. The complexity and dynamic changes of the environment are important factors affecting the phase control of smart reflective surfaces. In a dynamic and complex environment, long-term phase control is a difficult task. As a model-free algorithm, deep reinforcement learning can continuously adjust itself through feedback from environmental information to achieve an optimal strategy. In addition, the phase control of smart reflective surfaces involves a large number of reflective units, each of which has multiple adjustable phase angles, which leads to a very high-dimensional decision space. Traditional optimization methods are prone to fall into local optimal solutions in high-dimensional spaces, while deep reinforcement learning can effectively explore and utilize in high-dimensional spaces through policy networks and value networks to find the global optimal solution. Therefore, applying deep reinforcement learning to smart reflective surfaces will have a good effect. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and provide a method for adjusting the dynamic reflection coefficient of dual intelligent reflecting surfaces based on deep reinforcement learning.

[0005] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0006] A method for adjusting dynamic reflection coefficient of dual intelligent reflective surfaces based on deep reinforcement learning, comprising the following steps:

[0007] S1: Construct a wireless communication system model assisted by dual smart reflectors. All channel coefficients adopt the Rice channel model. Construct an optimization problem with the goal of maximizing the system communication rate. The constraints are the phase shift and amplitude reflection coefficient of the smart metasurface reflection module.

[0008] S2: Based on the transmission process of the smart metasurface smart reflective surface, the phase control process of the smart metasurface smart reflective surface is established as a Markov process;

[0009] S3: Solve the optimization problem based on the proximal strategy optimization algorithm, and accelerate the iteration rate through distributed computing through the distributed proximal strategy optimization algorithm to achieve dynamic adjustment of the reflection coefficient during the optimization solution process.

[0010] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0011] Preferably, in step S1, the wireless communication system model includes a base station, two smart metasurfaces, K single-antenna users and a controller with adjustable phase of the reflecting surface. The base station is equipped with M antennas and the number of reflecting units of the reflecting surface is N.

[0012] Preferably, in step S1, the optimization target is specifically:

[0013]

[0014] Where Θ1 represents the reflection coefficient matrix of RIS1; Θ2 represents the reflection coefficient matrix of RIS2; P sum represents the total data rate; r k represents the data rate of user k; K represents the number of users; SINR k represents the signal to interference and noise ratio of user k; Θ fs represents the reflection coefficient matrix of the first or second smart metasurface; diag(·) represents the diagonal matrix, j represents the imaginary unit, Respectively represent the phase shift of the 1st, ..., Nth reflection module of the first or second smart metasurface; Respectively represent the amplitude reflection coefficients of the 1st, ..., Nth reflection modules of the first or second smart metasurface; represents the phase shift of the nth reflection module of the first or second smart metasurface; Represents the amplitude reflection coefficient of the nth reflection module of the first or second smart metasurface.

[0015] Preferably, in step S3, the four-tuple is used represents a Markov process, represents the state space, represents the action space, represents the state transition function, represents the reward function; where the state space According to the elements that may affect the optimization target in the actual communication process, the state space The state set at time t is denoted as st = {H, Θ1, Θ2}, H = [h1,…,h K ] is the set of all user channels, h1,…,h K Represent the channels of users 1,…,K respectively; the action space The action set at time t is denoted as a t ={ΔΘ1, ΔΘ2}, ΔΘ1, ΔΘ2 represent the increments of the reflection coefficient matrices Θ1, Θ2 respectively; the reward function The functional form is:

[0016]

[0017] Among them, P t is the rate threshold.

[0018] Preferably, in step S4, the process of using the proximal strategy optimization algorithm to solve the optimization problem is as follows:

[0019] S41. In the initialization phase, the experience pool is initialized and configured with a capacity of C, the relevant parameters of the proximal strategy optimization algorithm are initialized, and the environment and agent of the wireless communication system are set;

[0020] S42. In each iteration of the loop phase, the number of channel estimations m and the number of state iterations n are obtained. First, it is determined whether the number of channel estimations m is 0:

[0021] If m = 0, the proximal strategy optimization algorithm ends;

[0022] If m≠0, continue to determine whether the number of state iterations n is 0: if n=0, set m=m-1, and then re-determine whether the number of channel estimations is 0; if n≠0, initialize state s0, and use the actor network to predict the state s at time j j Next, perform action a j The probability of using the critic network to predict the state s at time j j Under the evaluation value, the agent performs action a j , get the reward r at time j j and the state s at time j+1 j+1 , the state s at time j j 、Action a j , Reward j And the state s at time j+1 j+1 The four-tuple (s j ,a j ,r j ,s j+1 ) is stored in the experience pool, and the state s at time j is j Update to the state s at time +1 j+1, if the experience pool has not collected a four-tuple of length C and the loop has not ended, set n = n-1, and then re-determine whether the number of state iterations is 0; if the experience pool has collected a four-tuple of length C or the loop has ended, perform the following operations: calculate the advantage function G, calculate the cumulative discounted reward corresponding to each state, and update the actor network and the critic network;

[0023] S43. Continue iterating until the loop ends.

[0024] As a preferred option, the advantage function G at time t t The calculation method is as follows:

[0025] G t =δ t +(γλ)δ t+1 +…+(γλ) T-t+1 δ T-1

[0026] δ t =r t +γV(s t+1 )-V(s t )

[0027] Where γ is the discount factor; λ is the parameter in the generalized advantage estimation; δ t ,δ t+1 ,δ T-1 Respectively represent the intermediate variables at time t, t+1, and T-1; T represents the time parameter; V(s t+1 ),V(s t ) represent s t+1 ,s t The value of the state.

[0028] Preferably, the proximal policy optimization algorithm adopts the PPO-Clip method, and the objective function is to maximize the following objectives:

[0029]

[0030] in, Indicates that the policy parameter value is The value of the objective function when ; Represents the policy parameters; It means to take expectations; represents the probability ratio; clip represents the use of the PPO-clip method; ∈ represents a hyperparameter; π represents the strategy; are the strategy parameters before updating;

[0031] The specific way to use gradient ascent to update policy parameters is as follows:

[0032]

[0033] in, represents the updated policy parameters; α1 represents the learning rate; It means to find the gradient.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) Stronger adaptability to dynamic communication environments: As a model-free algorithm, deep reinforcement learning can continuously adjust itself through feedback from environmental information to achieve an optimal strategy, avoiding the problem that unified optimization methods are prone to falling into local optimal solutions in high-dimensional spaces.

[0036] (2) Faster algorithm iteration: The present invention uses the DPPO algorithm to accelerate the learning process by running multiple agent instances in parallel on multiple copies of the environment. This parallelization allows the system to generate more data in the same amount of time, allowing for faster policy updates. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of the steps of the method of the present invention;

[0038] Figure 2 It is a structural schematic diagram of a wireless communication system model assisted by dual intelligent reflective surfaces of the present invention;

[0039] Figure 3 A schematic diagram of the structure of the actor network and the critic network of the present invention;

[0040] Figure 4 A flow chart of a wireless communication system of the present invention;

[0041] Figure 5 A schematic diagram of the workflow of the distributed proximal strategy optimization algorithm of the present invention;

[0042] Figure 6 A sequence diagram of the configuration of the model-free intelligent reflective surface of the present invention;

[0043] Figure 7 A schematic diagram showing a comparison of performances of different algorithms in an embodiment of the present invention;

[0044] Figure 8 Schematic diagram of performance comparison of different algorithms under different numbers of reflection units in an embodiment of the present invention;

[0045] Fig. 9 is a histogram of average rates of different algorithms according to an embodiment of the present invention;

[0046] Fig.10 The result diagram of the bit error rate changing with SNR of the method of the present invention and the DDQN comparison method;

[0047] Fig.11 This is a schematic diagram showing the performance comparison of different algorithms when the number of users is 2 according to an embodiment of the present invention;

[0048] Fig.12 This is a schematic diagram showing the performance comparison of different algorithms when the number of users is 3 in an embodiment of the present invention;

[0049] Fig.13 This is a schematic diagram showing the performance comparison of different algorithms when the number of users is 4 in an embodiment of the present invention;

[0050] Fig.14 This is a schematic diagram showing the performance comparison of different algorithms when the number of users is 5 in an embodiment of the present invention;

[0051] Fig.15 It is a flow chart of the proximal strategy optimization algorithm of the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0053] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0054] This invention mainly proposes a method for adjusting the dynamic reflection coefficient of dual intelligent reflective surfaces based on deep reinforcement learning. This method establishes the entire intelligent reflective surface communication process in the form of a Markov chain, regulates the phase of the intelligent reflective surface through deep reinforcement learning, and improves the communication performance of the entire system in a dynamic environment through multiple rounds of iterative optimization. The specific implementation process is as follows:

[0055] like Figure 1 As shown, in a preferred implementation of the present invention, the above dual intelligent reflective surface dynamic reflection coefficient adjustment method based on deep reinforcement learning includes the following steps S1 to S4. The specific implementation process is described in detail below.

[0056] S1: Construct a wireless communication system model assisted by dual intelligent reflectors.

[0057] In step S1, Figure 2 As shown, the wireless communication system model includes a base station, two smart metasurface smart reflectors, K single-antenna users and a controller that can adjust the phase of the reflector. The base station is equipped with M antennas and the number of reflective units on the reflector is N.

[0058] For the wireless communication system, the first smart metasurface is denoted as RIS1, and the second smart metasurface is denoted as RIS2. The channel model between user k and the base station can be expressed as:

[0059]

[0060] Among them, h k represents the channel between user k and the base station; is the channel coefficient between the base station and user k; is the channel coefficient between the base station and RIS1; is the channel coefficient between RIS1 and RIS2; is the channel coefficient between RIS2 and user k; Θ1 represents the reflection coefficient matrix of RIS1, Θ2 represents the reflection coefficient matrix of RIS2, and the reflection coefficient matrix of RIS1 and the reflection coefficient matrix of RIS2 are unified by Θ fs Indicates that represents a diagonal matrix, j represents an imaginary unit, Respectively represent the phase shift of the 1st, ..., Nth reflection modules of RIS1 or RIS2; They represent the amplitude reflection coefficients of the 1st, ..., Nth reflection modules of RIS1 or RIS2 respectively.

[0061] Then, the base station sends a signal to the user, and the signal received by user k can be expressed as:

[0062]

[0063] Among them, y CEk is the part received by user k for channel estimation; y DTk is the part received by user k for information transmission; P s is the transmission power; a k is the power allocation coefficient of user k; α represents the proportion of the signal used for channel estimation, β represents the proportion of the signal used for information transmission, and α+β=1; x k is the signal sent to user k, and It means to take expectations; is subject to mean 0 and variance The Gaussian noise of Gaussian distribution. Therefore, the SINR of user k can be expressed as:

[0064]

[0065] Among them, SINR k represents the signal to interference and noise ratio SINR of user k; a t represents the power allocation coefficient of user t.

[0066] For a communication system, the performance metric may be SINR, data rate, frame error rate (FER), etc. Without loss of generality, the present invention adopts the total data rate as the performance metric, the total data rate P sum The calculation method is as follows:

[0067]

[0068] Among them, r k represents the data rate of user k.

[0069] In the wireless communication system model, all channel coefficients in the present invention consider the Rice channel model, with h Sk For example, the channel coefficient between the base station and user k can be expressed as:

[0070]

[0071] Among them, h Sk,LoS Indicates h Sk The line-of-sight (LoS) component, h Sk,NLoS Represents h Sk The non-line-of-sight (NLoS) component of the CNN is a Gaussian random variable with zero mean and unit variance, and both components are independent and identically distributed (iid) and circularly symmetric. In addition, the line-of-sight component is position-dependent and therefore slowly time-varying; the non-line-of-sight component is caused by multipath effects and therefore rapidly time-varying. It is the ratio of the power in the line-of-sight path to the power in the non-line-of-sight path.

[0072] S2: A wireless communication system model assisted by dual intelligent reflectors is constructed. All channel coefficients use the Rice channel model to construct an optimization problem with the optimization goal of maximizing the system communication rate. The constraints are the phase shift and amplitude reflection coefficient of the intelligent metasurface reflection module.

[0073] In step S2, the optimization target is specifically:

[0074]

[0075] S3: Based on the transmission process of the smart metasurface smart reflective surface, the phase control process of the smart metasurface smart reflective surface is established as a Markov process.

[0076] It should be noted that, in the present invention, generally speaking, some solutions to the optimal goal are usually solved by converting the problem into an alternating iterative optimization problem and using a related algorithm. However, in this wireless communication system model, the environment changes over time and is very complex, and the traditional model-based optimization algorithm cannot handle this type of optimization problem. To solve this technical problem, the present invention establishes a Markov process model for the phase control process of the intelligent reflector based on the traditional intelligent reflector transmission process. The entire control process of the controller for controlling the phase of the intelligent reflector does not affect the wireless communication system, and the wireless communication system has no other connection with the controller except for providing channel estimation information and phase information. The advantage of this design is that the controller can be deployed in other communication systems without modifying the content of any program. In deep reinforcement learning, an agent is an entity that can perceive the environment, make decisions and take actions. It continuously learns through interaction with the environment and gradually masters the optimal behavior strategy to achieve a specific goal. The agent of the present invention is an intelligent reflector control device, which can autonomously interact with the environment through an intelligent reflector to meet the design goals.

[0077] For S3, the established Markov process is as follows:

[0078] A Markov process can be represented by a quaternion To indicate that, represents the state space, represents the action space, represents the state transition function, represents the reward function, the state transfer function It is continuously optimized during training. In order to apply reinforcement learning to the configuration of intelligent reflective surface, the present invention first models the intelligent reflective surface assisted wireless communication as a Markov process. Specifically, the state space According to the elements that may affect the optimization target in the actual communication process, in the present invention, the state space The state set at time t is denoted as s t = {H, Θ1, Θ2}, H = [h1,…,h K ] is the set of all user channels, h1,…,h K denote the channels of users 1,…,K respectively. Considering that in a wireless communication system, the controller needs to adjust the phases of two smart reflectors, so the action space The action set at time t is recorded as the increment of the reflection coefficient matrices Θ1 and Θ2 of the two intelligent hypersurfaces, that is, a t={ΔΘ1,ΔΘ2}, ΔΘ1,ΔΘ2 represent the increments of the reflection coefficient matrices Θ1,Θ2 respectively. It should be noted that the reflection coefficient matrix has a range, so the increment also has a range limit. When the state s at time t t Execute action a at time t t After that, the wireless communication system will get a reward r at time t t , and the wireless communication system transfers to the state s at time t+1 t+1 .

[0079] Reward Function The functional form is:

[0080]

[0081] Among them, P t is the rate threshold.

[0082] When the total data rate P sum Less than the rate threshold P t When the total data rate P is , the present invention believes that this action has a greater impact on the performance of the wireless communication system, and the agent needs to be punished. The reason why it is set to 200 is that the punishment is not diluted by the daily reward. sum Greater than the rate threshold P t When the reward function is set, the agent is expected to not stay still or move all the time to “score points”, but to strive to obtain a higher total data rate P. sum .

[0083] S4: Solve the optimization problem based on the Proximal Policy Optimization (PPO) algorithm, and accelerate the iteration rate through distributed computing through the Distributed Proximal Policy Optimization (DPPO) algorithm, so as to realize dynamic adjustment of the reflection coefficient in the optimization solution process.

[0084] It should be noted that in the present invention, the proximal policy optimization algorithm PPO is an algorithm widely used in the field of reinforcement learning, which was proposed by Schulman et al. in 2017. The proximal policy optimization algorithm PPO aims to solve some problems of the policy gradient method, such as sample efficiency and ease of implementation. The core idea is to limit the step size of the policy change when the policy is updated to avoid performance collapse caused by excessive updates. The proximal policy optimization algorithm PPO attempts to solve these problems by limiting the difference between the new policy and the old policy. The algorithm introduces a clip function to prevent the new policy from deviating too far from the old policy. Therefore, the advantage function G at time t t The calculation of is as follows:

[0085] G t =δ t +(γλ)δ t+1 +…+(γλ) T-t+1 δ T-1

[0086] δ t =r t +γV(s t+1 )-V(s t )

[0087] Among them, γ is the discount factor; λ is the parameter in Generalized Advantage Estimation (GAE); δ t ,δ t+1 ,δ T-1 Respectively represent the intermediate variables at time t, t+1, and T-1; T represents the time parameter; V(s t+1 ),V(s t ) represent s t+1 ,s t The value of the state.

[0088] The advantage function tells the invention which actions are better than the average and therefore should be chosen more often when learning. By applying gradient ascent to those actions with positive advantage values, the policy can be biased towards better actions.

[0089] In the proximal policy optimization algorithm (PPO), the pace of policy updates is limited to ensure that the new policy does not deviate too far from the old policy. The use of the advantage function is consistent with this goal because it promotes the stability of updates by emphasizing actions that are better relative to the average behavior rather than all possible actions. The advantage function can encourage the proximal policy optimization algorithm to explore actions that may not be the current optimal but have potential. If the advantage value of an action is positive, the proximal policy optimization algorithm will increase the probability of selecting it even if it is not the optimal action, thereby achieving effective exploration.

[0090] The definition of the objective function L is also very important. The proximal policy optimization algorithm PPO defines a special objective function. According to the different optimization methods it adopts, it can be divided into two types: one is the objective function using the PPO-Clip (Proximal Policy Optimization with Clip) method, and the other is the objective function using the PPO-Penalty method. The present invention adopts the PPO-Clip method. For PPO-Clip, the objective function is to maximize the following objectives:

[0091]

[0092] in, Indicates that the policy parameter value is The value of the objective function when ; Represents the policy parameters; It means to take expectations; represents the probability ratio; clip represents the use of the PPO-clip method; ∈ represents a hyperparameter. In this embodiment, the hyperparameter ∈ is set to between 0.1 and 0.3; π represents the strategy; are the policy parameters before updating.

[0093] PPO-Clip uses a clip function to prevent the policy update from deviating too far, which is done by limiting the policy ratio r t (θ) to the interval [1-∈, 1+∈], as described above. This clip function ensures that the policy update will not be too large, even if the advantage function G t This is also true in most cases. Finally, the policy update function is set up, using gradient ascent to update the policy parameters:

[0094]

[0095] in, represents the updated policy parameters; α1 represents the learning rate; It means to find the gradient.

[0096] In this way, the proximal policy optimization algorithm allows small policy updates, avoiding the performance instability that may be caused by large updates. In practice, the proximal policy optimization algorithm PPO has been proven to achieve good results on a variety of tasks, so it is one of the most popular reinforcement learning algorithms.

[0097] In addition, the proximal policy optimization algorithm is a reinforcement learning algorithm that uses the Actor-Critic architecture. In the Actor-Critic method, there are two main networks, the actor network Actor and the critic network Critic. The Actor network is responsible for learning the policy, that is, given a state, deciding what action to take. In PPO, this is usually implemented through a neural network that outputs the probability of each possible action in a given state (in discrete action space) or the parameters of the action (in continuous action space). The Critic network evaluates the quality of the current strategy, usually by estimating the state value function V(s), that is, starting from the current state, the expected reward that can be obtained by following the current strategy.

[0098] The proximal policy optimization algorithm PPO optimizes the policy through the Actor network and uses the Critic network to reduce the variance of the policy gradient estimation, which can accelerate the learning process and improve the stability of learning. The core of the proximal policy optimization algorithm is to limit the step size of the policy update to ensure that the updated policy does not deviate too far from the original policy, thereby avoiding drastic fluctuations in performance during training. For the wireless communication system assisted by dual intelligent reflectors, the Actor network and the Critic network are designed as follows: Figure 3 shown.

[0099] After a basic introduction to the proximal strategy optimization algorithm, the actor network, and the critic network, the present invention will introduce the basic framework of the entire dual intelligent reflector-assisted wireless communication system process. Figure 4 As shown, the optimization goal of the present invention is to solve the optimization problem in step S2. First, at time t, the state of the environment is s t , in order to explore the t The optimal strategy under , using the actor network for prediction, obtains the mean and variance of the distribution of the reflection coefficient increment:

[0100]

[0101] Among them, μ1(t), μ2(t) represent the mean of the incremental distribution of RIS1 and RIS2 reflection coefficients at time t; σ1(t), σ2(t) represent the variance of the incremental distribution of RIS1 and RIS2 reflection coefficients at time t, W represents the actor network, Represents the actor network parameters.

[0102] After sampling the actor network, we can get an increment a t ={ΔΘ1(t),ΔΘ2(t)}. The intelligent reflective surface modifies the reflection coefficient through the controller to obtain:

[0103] Θ1(t+1)=Θ1(t)⊙ΔΘ1(t)

[0104] Θ2(t+1)=Θ2(t)⊙ΔΘ2(t)

[0105] Among them, Θ1(t+1) represents the reflection coefficient of RIS1 at time t+1; Θ1(t) represents the reflection coefficient of RIS1 at time t; ΔΘ1(t) represents the increment of the reflection coefficient of RIS1 at time t; Θ2(t+1) represents the reflection coefficient of RIS2 at time t+1; Θ2(t) represents the reflection coefficient of RIS2 at time t; ΔΘ2(t) represents the increment of the reflection coefficient of RIS2 at time t; ⊙ is the Hadamard product.

[0106] The wireless communication system then performs the next round of signal transmission and obtains the channel estimation value H(t+1) and the instant reward r t, H(t+1) and Θ1(t+1), Θ2(t+1) together constitute the state s at the next moment t+1 ,(s t ,a t ,r t ,s t+1 ) is stored as a four-tuple in the memory unit and used as an element for subsequent actor network and critic network updates. Then, s t+1 Substitute into the actor network to start the next cycle.

[0107] The actor network and the critic network are updated using the existing experience replay mechanism. When the four-tuple in the memory unit reaches the capacity C, some four-tuples are taken out to update the actor network and the critic network. l ,a l ,r l ,s l+1 ), first according to the actor network W of the current moment u u and network parameters Calculate the mean and variance of the output action, recorded as:

[0108]

[0109] in, Represents the incremental distribution of reflection coefficients of RIS1 and RIS2 at time l, respectively, in the actor network W u and network parameters Calculate the new mean value; Represents the incremental distribution of RIS1 and RIS2 reflection coefficients at time l, in the actor network W u and network parameters Calculate the new variance; s l ,a l ,r l ,s l+1 They represent the state, action, reward at time l and the state at time l+1 respectively.

[0110] Correspondingly, the reflection coefficient increments of RIS1 and RIS2 are distributed in the actor network W and the network parameters The distribution of is old, that is:

[0111]

[0112] in, Respectively represent the distribution of RIS1 and RIS2 reflection coefficient increments in the actor network W and network parameters The mean value before the update at time l under Respectively represent the distribution of RIS1 and RIS2 reflection coefficient increments in the actor network W and network parameters The variance before the update calculated at time l under the condition of ; μ1(l),μ2(l) represent the distribution of the incremental reflection coefficients of RIS1 and RIS2 in the actor network W and the network parameters respectively The value of the old mean at time l under ; σ1(l), σ2(l) represent the distribution of the incremental reflection coefficients of RIS1 and RIS2 in the actor network W and the network parameters respectively. The value of the old variance calculated at time l under .

[0113] Then calculate the advantage function G t And the objective function Finally update the actor network and critic network.

[0114] Secondly, the critic network is adapted to generate an action-value function V(·) to evaluate whether the actor network’s strategy is correct. The critic network is updated by minimizing the loss function, which is defined as

[0115]

[0116] in, Indicates that the policy parameter value is The value of the objective function when Q u represents the actor network index; In the actor network Q u The following network parameters.

[0117] Finally, the policy update function is set up, using gradient ascent to update the policy parameters, namely:

[0118]

[0119] in, They represent the updated strategy parameters under the actor network Q and the strategy parameters before updating under the actor network Q respectively; Indicates that the policy parameter value under the actor network Q is is the gradient value of the objective function; α2 is the learning rate.

[0120] According to the above discussion, the process of solving the optimization problem by the proximal strategy optimization algorithm proposed in step S4 of the present invention is as follows: Fig.15 As shown, the details are as follows:

[0121] S41. In the initialization phase, the experience pool is initialized and configured with a capacity of C, the relevant parameters of the proximal strategy optimization algorithm are initialized, and the environment and agent of the wireless communication system are set;

[0122] S42. In each iteration of the loop phase, the number of channel estimations m and the number of state iterations n are obtained. First, it is determined whether the number of channel estimations m is 0:

[0123] If m = 0, the proximal strategy optimization algorithm ends;

[0124] If m≠0, continue to determine whether the number of state iterations n is 0: if n=0, set m=m-1, and then re-determine whether the number of channel estimations is 0; if n≠0, initialize state s0, and use the actor network to predict the state s at time j j Next, perform action a j The probability of using the critic network to predict the state s at time j j Under the evaluation value, the agent performs action a j , get the reward r at time j j and the state s at time j+1 j+1 , the state s at time j j 、Action a j , Reward j And the state s at time j+1 j+1 The four-tuple (s j ,a j ,r j ,s j+1 ) is stored in the experience pool, and the state s at time j is j Update to the state s at time +1 j+1 , if the experience pool has not collected a four-tuple of length C and the loop has not ended, set n = n-1, and then re-determine whether the number of state iterations is 0; if the experience pool has collected a four-tuple of length C or the loop has ended, perform the following operations: calculate the advantage function G, calculate the cumulative discounted reward corresponding to each state, and update the actor network and the critic network;

[0125] S43. Continue iterating until the loop ends.

[0126] It should be noted that, in step S4 of the present invention, in order to speed up the learning process of the entire wireless communication system, the present invention adopts a distributed proximal policy optimization (DPPO) algorithm to accelerate the learning process by running multiple agent instances in parallel on multiple environment copies. This parallelization allows the wireless communication system to generate more data in the same time, so that the policy can be updated faster.

[0127] Specifically, if Figure 5As shown in the figure, the main reinforcement learning model PPO is running in the main process of the DPPO algorithm. In this system, there are multiple sub-processes, and sub-process 1, sub-process 2, and sub-process 3 can be responsible for different tasks respectively. Among them, each sub-process will collect experience data, and these experience data will be passed to the PPO model in the main process. The PPO model in the main process will be updated according to these experiences, and the updated model will continue to affect the operation of the sub-process. "net" and "GPU" indicate that the network structure and graphics processor play an important role in this system, such as performing model calculations and acceleration. The entire architecture presents a distributed feature, and each sub-process cooperates with the main process to jointly promote the progress of reinforcement learning tasks.

[0128] The following will explain how parallelization helps speed up the learning process and how the DPPO algorithm is combined with the PPO algorithm.

[0129] In traditional PPO, an agent interacts with an environment to generate sequence data s t ,a t ,r t ,s t+1 In DPPO, it is assumed that parallel workers, each of which can independently generate such sequences in its own copy of the environment. Thus, if each worker runs time step, the system can generate a total of data points. This is compared to a single agent which can only generate data points, achieving significant speedup.

[0130] In a distributed setting, the policy parameters After the update, these updates need to be synchronized to all workers. This can be done with a central server that collects the gradients and updates the parameters, and then broadcasts the updated parameters back to the workers.

[0131] Therefore, by generating a large amount of data in parallel and updating the policy more frequently, DPPO is able to achieve faster convergence speed during the learning process. This acceleration effect can be briefly summarized by the following formula:

[0132]

[0133]

[0134] in, is the number of parallel workers, is the number of time steps collected by each worker.

[0135] It should be noted that the efficiency of distributed systems does not always scale linearly, as it may be affected by factors such as network bandwidth, communication delays, and synchronization mechanisms. Therefore, the actual efficiency of DPPO is improved by using multiple parallel workers to accelerate the learning process, each of which collects data independently in its own copy of the environment. This approach greatly increases data throughput and sample diversity relative to a single process, and can be effectively scaled on multi-core or multi-machine distributed computing resources. The following invention will introduce how DPPO accelerates learning.

[0136] In DPPO, each worker process performs the following operations:

[0137] 1) Interact with the environment independently

[0138] 2) Collect a set of transformations s,a,r,s′

[0139] 3) Calculate the gradient of this data

[0140] If you have worker processes, you can collect times more data, thus speeding up learning. The settings of policy gradient estimation and objective function are the same as those introduced above.

[0141] During synchronization and updating, each worker process independently computes gradients, which are then sent to the central parameter server. The server updates the policy parameters The updated parameters are then broadcast back to each worker process. The mathematical expression of this process can be simplified to:

[0142]

[0143] in, Respectively represent the values ​​of the strategy parameters after the update and the values ​​of the strategy parameters before the update; is the number of worker processes; α3 is the learning rate; is the cutting objective function calculated by the i-th worker process.

[0144] In this way, DPPO is able to accumulate large amounts of data across multiple worker processes, quickly iterate and update policies, and ultimately accelerate the learning process. This approach is particularly useful for complex environments and tasks where a single agent may require a large number of interactions to learn an effective policy.

[0145] It should be noted that if Figure 6As shown, a cycle in the optimization process of the reflection coefficient of the intelligent reflection surface is shown. After the intelligent reflection surface updates the reflection coefficient according to the instructions of the intelligent agent, the wireless communication system performs signal transmission and channel estimation. Then the base station BS and the user UE transmit the reward information (Reward) and state information (State) to the controller respectively. Finally, the controller updates the network according to the information transmitted back.

[0146] In order to better demonstrate the specific implementation and technical effects of the present invention, the dynamic reflection coefficient adjustment method of dual intelligent reflective surfaces based on deep reinforcement learning shown in steps S1 to S4 in the above preferred implementation method is applied to a specific example.

[0147] Example

[0148] The specific implementation process of the dynamic reflection coefficient adjustment method of dual intelligent reflective surfaces based on deep reinforcement learning adopted in this embodiment is as described above and will not be repeated here.

[0149] In this embodiment, some parameters are simulated to verify the effectiveness of the method of the present invention. The experimental settings of this embodiment are as follows: the number of BS antennas is set to 2, the number of reflection units of the intelligent reflection surface is set to 36, the number of users is set to 2, and each user is equipped with an antenna. The location of the intelligent reflection surface is set to (-3,5,5), the location of the BS is set to (0,0,10), and the users are randomly distributed in a square with a side length of 10, and the height is set to 0. Discount factor γ = 0.99, learning rate α = 0.0003, GAE parameter λ = 0.95, and hyperparameter ε = 0.2. In the simulation experiment, this embodiment uses three comparison methods for comparison, namely DDQN, MAB, and Random. Among them, the Random group uses completely random phase selection, MAB is to fix the reflection module with the best average performance for reflection, and DDQN means that the DDQN algorithm with a discrete action set is used to optimize the reflection coefficient.

[0150] Figure 7 This is a comparison chart of different algorithms, where the x-axis is the number of iterations and the y-axis is P m The moving average of the window length is 100. The value on the right side of the line graph represents the mean value after the four methods reach convergence. It can be seen that the method proposed in this invention ( Figure 7DPPO) is better than the compared solutions in terms of iteration speed and final performance. The random phase method does not rely on any external information and is undoubtedly the worst. MAB calculates the average performance of each reflection unit and selects the best one, which slightly improves the performance. DDQN, which also uses deep reinforcement learning methods, and the method of the present invention can far exceed the Random method and MAB method after several iterations, because the former two will update the reflection coefficient from the feedback information of the environment for iterative optimization. Figure 7 It can be seen that the method of the present invention can reach a relatively stable value in about 100 rounds, while DDQN needs about 200 rounds to reach a stable value. This may be affected by many factors, but the biggest influencing factor is that the method of the present invention adopts a distributed training process, which greatly reduces the number of rounds required to reach convergence. Secondly, it may also be related to the setting of the reward function or the choice of algorithm. Finally, when the method of the present invention and DDQN reach a relatively stable value, the final convergence value of the method of the present invention is higher than that of DDQN. The reason is that the method of the present invention is a continuous action set, while DDQN is a discrete action set. The present invention is more likely to reach the theoretical optimal value.

[0151] This embodiment also compares the above algorithm and the method of the present invention under different numbers of reflection units (reflection modules). Figure 8 and Fig. 9 As shown, similarly, the x-axis in the figure is the number of iterations, and the y-axis is P m The moving average of . Figure 8 The performance of the above four algorithms is shown when the reflection units are 36 and 16 respectively. Figure 8 It can be seen from the figure that when the number of reflection units decreases, the performance of the algorithm decreases by about 15% to 20%. Correspondingly, the gap between the algorithms will also increase with the increase of reflection units, but the method of the present invention is always better than the other three comparison methods. Fig. 9 The comparison of the average rates of the above four algorithms under different numbers of reflection units is shown. It can be seen that the average rates of the above four algorithms increase with the increase of the number of reflection surface units, but the rate of increase slows down and eventually tends to a constant value, and the method of the present invention is always better than the other three comparison methods.

[0152] Fig.10 The BER variation of the DDQN method and the method of the present invention is compared with that of the method of the present invention. The present invention uses the phase information after 2000 iterations as the parameter for testing the BER, and the user position is randomized. Fig.10 It can be seen that the performance of the method of the present invention is better than that of DDQN. Through simulation experiments, it is verified that the method of the present invention based on DPPO can improve the communication performance in a long time.

[0153] Figure 11-Figure 14 The performance of the above four different algorithms is shown when the number of users is 2, 3, 4 and 5 respectively. Figure 11-Figure 14 The x-axis is the number of iterations and the y-axis is P m It can be seen that no matter how many users there are, the method of the present invention is always better than the other three comparison methods. In addition, it can be seen that with the increase of the number of users, the overall performance of each algorithm is reduced, and the stability of the algorithm is also reduced with the increase of the number of users. However, the number of iterations to reach convergence does not change significantly, which shows that the change in the number of users does not affect the speed of algorithm convergence.

[0154] In general, the present invention proposes a method for adjusting the dynamic reflection coefficient of dual intelligent reflective surfaces based on deep reinforcement learning for a wireless communication system with dual intelligent reflective surfaces. In the method of the present invention, how to establish an optimization model for the reflection coefficient of an intelligent reflective surface suitable for a dynamic environment is studied, which will involve considering factors such as dynamic changes in the environment, multipath propagation, interference and noise, and incorporating them into the model. The present invention will use deep reinforcement learning as the basis of the optimization algorithm, and study how to design a reinforcement learning framework suitable for the optimization of the reflection coefficient of the intelligent reflective surface, including state representation, action selection, reward design and value function estimation. The present invention designs a reasonable environmental interaction mechanism so that the intelligent reflective surface reflection coefficient optimization algorithm can interact with the dynamic environment. Through interaction with the environment, iterative updates of the strategy will be achieved to gradually optimize the configuration of the reflection coefficient of the intelligent reflective surface. This will include steps such as selecting appropriate actions, evaluating feedback information and updating strategy parameters.

[0155] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A method for adjusting dynamic reflection coefficient of dual intelligent reflective surfaces based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Construct a wireless communication system model assisted by dual smart reflectors. All channel coefficients adopt the Rice channel model. Construct an optimization problem with the goal of maximizing the system communication rate. The constraints are the phase shift and amplitude reflection coefficient of the smart metasurface reflection module. S2: Based on the transmission process of the smart metasurface smart reflective surface, the phase control process of the smart metasurface smart reflective surface is established as a Markov process; S3: Solve the optimization problem based on the proximal strategy optimization algorithm, and accelerate the iteration rate through distributed computing through the distributed proximal strategy optimization algorithm to achieve dynamic adjustment of the reflection coefficient during the optimization solution process.

2. A method for adjusting dynamic reflection coefficients of dual intelligent reflective surfaces based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S1, the wireless communication system model includes a base station, two smart metasurfaces, K single-antenna users, and a controller that can adjust the phase of the reflection surface. The base station is equipped with M antennas, and the number of reflection units on the reflection surface is N.

3. A method for adjusting dynamic reflection coefficients of dual intelligent reflective surfaces based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S1, the optimization goal is specifically: Where Θ1 represents the reflection coefficient matrix of RIS1; Θ2 represents the reflection coefficient matrix of RIS2; P sum represents the total data rate; r k represents the data rate of user k; K represents the number of users; SINR k represents the signal to interference and noise ratio of user k; Θ fs represents the reflection coefficient matrix of the first or second smart metasurface; diag(·) represents the diagonal matrix, j represents the imaginary unit, Respectively represent the phase shift of the 1st, ..., Nth reflection module of the first or second smart metasurface; Respectively represent the amplitude reflection coefficients of the 1st, ..., Nth reflection modules of the first or second smart metasurface; represents the phase shift of the nth reflection module of the first or second smart metasurface; Represents the amplitude reflection coefficient of the nth reflection module of the first or second smart metasurface.

4. A method for adjusting dynamic reflection coefficients of dual intelligent reflective surfaces based on deep reinforcement learning as claimed in claim 3, characterized in that: In step S3, the four-tuple is used represents a Markov process, represents the state space, represents the action space, represents the state transition function, represents the reward function; where the state space According to the elements that may affect the optimization target in the actual communication process, the state space The state set at time t is denoted as s t = {H, Θ1, Θ2}, H = [h1,…,h K ] is the set of all user channels, h1,…,h K Represent the channels of users 1,…,K respectively; the action space The action set at time t is denoted as a t ={ΔΘ1, ΔΘ2}, ΔΘ1, ΔΘ2 represent the increments of the reflection coefficient matrices Θ1, Θ2 respectively; the reward function The functional form is: Among them, P t is the rate threshold.

5. The method for adjusting dynamic reflection coefficient of dual intelligent reflective surfaces based on deep reinforcement learning according to claim 1, characterized in that: In step S4, the process of using the proximal strategy optimization algorithm to solve the optimization problem is as follows: S41. In the initialization phase, the experience pool is initialized and the capacity is configured as C, the relevant parameters of the proximal strategy optimization algorithm are initialized, and the environment and intelligent agent of the wireless communication system are set; S42. In each iteration of the loop phase, the number of channel estimations m and the number of state iterations n are obtained. First, it is determined whether the number of channel estimations m is 0: if m = 0, the proximal strategy optimization algorithm ends; if m≠0, it is continued to determine whether the number of state iterations n is 0: if n = 0, m = m-1 is set, and then it is re-determined whether the number of channel estimations is 0; If n≠0, initialize state s0 and use the actor network to predict the state s at time j j Next, perform action a j The probability of using the critic network to predict the state s at time j j Under the evaluation value, the agent performs action a j , get the reward r at time j j and the state s at time j+1 j+1 , the state s at time j j 、Action a j , Reward j And the state s at time j+1 j+1 The four-tuple (s j ,a j ,r j ,s j+1 ) is stored in the experience pool, and the state s at time j is j Update to the state s at time +1 j+1 , if the experience pool has not collected a four-tuple of length C and the loop has not ended, set n = n-1, and then re-determine whether the number of state iterations is 0; if the experience pool has collected a four-tuple of length C or the loop has ended, perform the following operations: calculate the advantage function G, calculate the cumulative discounted reward corresponding to each state, and update the actor network and the critic network; S43. Continue iterating until the loop ends.

6. A method for adjusting dynamic reflection coefficients of dual intelligent reflective surfaces based on deep reinforcement learning as claimed in claim 5, characterized in that: Advantage function G at time t t The calculation method is as follows: G t =d t +(gl)d t+1 +…+(cl) T-t+1 d T-1 δ t =r t +γV(s t+1 )-V(s t ) Where γ is the discount factor; λ is the parameter in the generalized advantage estimation; δ t ,δ t+1 ,δ T-1 Respectively represent the intermediate variables at time t, t+1, and T-1; T represents the time parameter; V(s t+1 ),V(s t ) represent s t+1 ,s t The value of the state.

7. A method for adjusting dynamic reflection coefficients of dual intelligent reflective surfaces based on deep reinforcement learning as claimed in claim 6, characterized in that: The proximal policy optimization algorithm adopts the PPO-Clip method, and the objective function is to maximize the following objectives: in, Indicates that the policy parameter value is The value of the objective function when ; Represents the policy parameters; It means to take expectations; represents the probability ratio; clip represents the use of the PPO-clip method; ∈ represents a hyperparameter; π represents the strategy; are the strategy parameters before updating; The specific way to use gradient ascent to update policy parameters is as follows: in, represents the updated policy parameters; α1 represents the learning rate; It means to find the gradient.

Images