Reconfigurable intelligent surface beam forming method based on model-driven reinforcement learning

Abstract

The invention discloses a reconfigurable intelligent surface beam forming method based on model-driven reinforcement learning, and the method comprises the steps: initializing an experience playback memory, training action subnet parameters, target action subnet parameters, training evaluation subnet parameters and target evaluation subnet parameters of reinforcement learning before deployment; after a channel estimation result is obtained, initializing a passive beamforming vector of the RIS; initializing an active beam forming vector; initializing the state of the reinforcement learning network; during iteration, the training action subnet generates a corresponding action according to the current state, calculates an award and updates the state of the intelligent agent; storing the experience in a playback memory; when the algorithm converges, outputting an active beamforming vector and a passive beamforming vector; and extracting experience from the playback memory, updating the training action subnet and the training evaluation subnet by minimizing a loss function, and updating the target action subnet and the target evaluation subnet by using soft update. The method is good in performance, low in complexity and high in adaptive capacity.

Description

Technical Field

[0001] The invention relates to communication technology, and in particular to a reconfigurable intelligent surface beamforming method based on model-driven reinforcement learning. Background Art

[0002] The development of communication technology has triggered the demand for higher data rates and higher coverage. Reconfigurable Intelligent Surfaces (RIS) can customize the communication environment according to specific goals by controlling the reflection, refraction and absorption characteristics of electromagnetic waves, becoming a potential way to meet these needs. In this context, the joint beamforming design in RIS-assisted systems has received great attention, but most of the existing work focuses on single RIS-assisted systems, and research on multi-RIS-assisted systems is still relatively lacking.

[0003] In recent years, the application of deep learning (DL) technology in physical layer communications has increased. Existing DL-based joint beamforming work mainly focuses on single RIS-assisted systems, relying on data-driven methods, performing poorly in dynamic environments and lacking interpretability. In contrast, model-driven DL methods improve resilience and interpretability by integrating expert knowledge, but existing model-driven DL architectures have problems with layer-dependent parameters and non-adaptive iterations, which limits their application in dynamic scenarios. Deep reinforcement learning (DRL) technology has also been applied to joint beamforming research. DRL collects data and self-trains autonomously through continuous interaction with the environment. However, the DRL method faces the challenges of a large number of trainable variables and a slow convergence process. It is expected to combine the model-driven method with DRL to propose an innovative framework suitable for wireless communication applications, which can alleviate the complexity of network training and improve performance by integrating expert knowledge. Summary of the invention

[0004] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a reconfigurable intelligent surface beamforming method based on model-driven reinforcement learning.

[0005] Technical solution: The RIS-assisted system joint beamforming method of model-driven reinforcement learning described in the present invention includes:

[0006] (1) Before deployment, the reinforcement learning experience replay memory, training action subnet parameters, target action subnet parameters, training evaluation subnet parameters, and target evaluation subnet parameters are randomly initialized.

[0007] (2) After obtaining the channel estimation result, the passive beamforming vector of RIS is randomly initialized; the active beamforming vector is initialized using the zero forcing algorithm; and the state of the reinforcement learning network is initialized.

[0008] (3) During iteration, the training action subnet generates corresponding actions based on the current state, calculates rewards and updates the agent state; stores the experience in the replay memory; and outputs the active beamforming vector and the passive beamforming vector when the algorithm converges.

[0009] (4) Extract a batch of experiences from the experience replay memory, update the training action subnet and the training evaluation subnet by minimizing the loss function, and use soft update to update the target action subnet and the target evaluation subnet.

[0010] Further, step (2) specifically includes:

[0011] (2.1) Randomly initialize the passive beamforming vector and

[0012] (2.2) Calculate the active beamforming vector W using the ZF algorithm (0) =H H (HH H ) -1 , where H = [h1,h2,…,h K ] H is the matrix composed of each user combined channel, K is the number of users, h k is the combined channel of user k;

[0013] (2.3) Initialize the state of the reinforcement learning network H all They represent the achievable rate and corresponding to the initialization beamforming vector, the change in the achievable rate and between two consecutive iterations, the change in the active beamforming vector between two consecutive iterations, the action of the agent at initialization, and the matrix composed of all channels in the system. Optimize the passive beamforming vector when initializing separately and The step length.

[0014] Furthermore, step (3) specifically includes:

[0015] (3.1) The action subnet generates the algorithm step size based on the observation and The observation is represented by t represents the time, s (t) [1:6] indicates state s(t) The first 6 elements of ;

[0016] (3.2) Calculate the auxiliary intermediate variables u and λ. The elements in u and λ are recorded as:

[0017]

[0018] P T is the maximum transmit power, w i is the active beamforming vector of the ith user, is the receiving noise variance of user k, u k is the receiving coefficient of user k;

[0019] (3.3) Update the active beamforming vector according to the auxiliary intermediate variables u and λ calculated in step (3.2);

[0020]

[0021] α k is the priority of user k;

[0022] (3.4) Calculate the auxiliary intermediate variables u and λ again;

[0023] (3.5) Using the step size output by the action subnet The passive beamforming vector of the first reconfigurable smart surface is updated and recorded as

[0024]

[0025] in is the value of the reflection phase φ1 of the first reconfigurable smart surface in the previous iteration,

[0026]

[0027] Where G1 and G2 are the channels from AP to the first reconfigurable smart surface and from AP to the second reconfigurable smart surface respectively; h 1,2 is the channel between two reconfigurable smart surfaces; h 1,k 、h 2,k are the channels from the first reconfigurable smart surface to user k and the channels from the second reconfigurable smart surface to user k, respectively; R{·} represents the real part of the variable, J is the imaginary unit, and φ2 is the reflection phase of the first reconfigurable smart surface;

[0028] (3.6) The third calculation of auxiliary intermediate variables u and λ;

[0029] (3.7) Using the step size output by the action subnet updating the passive beamforming vector of the second reconfigurable smart surface;

[0030] (3.8) Calculate the current reward

[0031]

[0032] where r con is the reward for algorithm convergence, r iter is the penalty for each iteration of the algorithm, They represent the change of the achievable rate and the active beamforming vector in the t-th iteration, the change of the achievable rate and the convergence judgment threshold, and the active beamforming vector convergence judgment threshold respectively;

[0033] (3.9) Update the agent status in Represents the change in the sum of the achievable rates of two consecutive iterations, represents the change of active beamforming vector between two consecutive iterations, H all =[Vec(G1),Vec(G2),Vec(H1),Vec(H2),Vec(H 1,2 )] contains the real form of all channel matrices, where H1 = [h 1,1 ,…,h 1,K ] and H2=[h 2,1 ,…,h 2,K ],Vec(A)=[R(a 1,1 ,…,a 1,N ,…,a N,1 ,…,a N,N ),I(a 1,1 ,…,a 1,N ,…,a N,1 ,…,a N,N )] represents the operation of concatenating the real and imaginary parts of a matrix and converting them into a vector. i,j Represents the element in the i-th row and j-th column of the matrix;

[0034] (3.10) Update agent experience {s (t) ,a (t) ,r (t) ,s (t+1)}stored in playback memory;

[0035] (3.11) satisfies the convergence condition and When , the current active beamforming vector and passive beamforming vector are output.

[0036] Further, step (4) specifically includes:

[0037] (4.1) Extract a batch of experience s from the experience replay memory(p) ,a (p) ,r (p) ,s (p+1) ;

[0038] (4.2) Update the training evaluation subnet by minimizing the loss function

[0039]

[0040] (4.3) Update the training action subnet by minimizing the loss function

[0041]

[0042] (4.4) Use soft update to update the target action subnet and target evaluation subnet, denoted as

[0043]

[0044] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0045] The present invention rationally combines the model-driven concept with DRL. In the RIS-assisted system, the achievable rate performance is significantly improved and the algorithm complexity is greatly reduced. The present invention proposes a model-driven reinforcement learning network, which greatly reduces the number of network training parameters and greatly shortens the training time. The number of network iterations can be adjusted according to different environmental dynamics, and the network can be adapted through online learning when the deployment environment changes. In general, the present invention provides a RIS-assisted system joint beamforming solution with practical deployment prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of an embodiment of the present invention;

[0047] Figure 2 It is a structural diagram of the action subnet of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be described in detail below with reference to the accompanying drawings and an example of a dual RIS-assisted downlink multi-user MISO system.

[0049] 1. Channel Model Applicable to This Embodiment

[0050] In a MU-MISO system with one AP and K single-antenna users, two RISs are introduced to assist the communication process of the system. In this embodiment, the number of users is K = 4, and both RIS1 and RIS2 have N1 = N2 = 40 reflection elements. The maximum transmit power of the AP is P T=9dBm, the carrier frequency is 3GHz, and the distance between adjacent antennas is half a wavelength. The channel from AP to RIS1, the channel from AP to RIS2, the channel from RIS1 to user k, the channel from RIS2 to user k, and the channel from RIS1 to RIS2 are represented as G1, G2, h 1,k 、h 2,k and H 1,2 The phase shift introduced by the nth reflection unit of the lth RIS is expressed as φ l,n ∈[0,2π), record The corresponding reflection coefficient is expressed as where n∈1,2,…,N l , l∈1,2. The passive beamforming vector of RISl is denoted as N l is the number of reflection units of RIS1. The channel between AP and user can be expressed as:

[0051]

[0052] in, represents the phase shift matrix associated with RIS1.

[0053] The signal transmitted to user k is denoted as s k , in this embodiment, is assumed to be an independent random variable with a mean of zero and a variance of unity. The AP first uses the active beamforming vector w k Processes the transmission signal sent to user k. This vector is constrained The limit of P T represents the maximum transmission power. After channel transmission, the signal received by user k is expressed as

[0054]

[0055] in represents the received noise at user k. When the channel CSI is known, the received signal-to-noise ratio (SINR) can be expressed as:

[0056]

[0057] When the priority of user k is denoted as α k When , the user’s achievable rate is expressed as

[0058]

[0059] 2. Specific steps of this embodiment

[0060] like Figure 1As shown in the figure, model-driven reinforcement learning is applied to wireless communication design and used for joint beamforming design of RIS system. The complete joint beamforming process includes three steps: initialization, algorithm iteration, and network update:

[0061] (1) Initialization

[0062] Before the algorithm is deployed, the experience replay memory of reinforcement learning is randomly initialized to train the action subnet parameters Target Action Subnet Parameters Training and evaluation subnet parameters Target evaluation subnet parameters

[0063] During the deployment process, each time after obtaining CSI, the passive beamforming vector of RIS is randomly initialized. and Using the ZF algorithm to calculate the active beamforming vector W (0) =H H (HH H ) -1 , where H = [h1,h2,…,h K ] H , K is the number of users, h k is the combined channel of user k. Initialize the state of the reinforcement learning network They represent the achievable rate and corresponding to the initialization beamforming vector, the change in the achievable rate and between two consecutive iterations, the change in the active beamforming vector between two consecutive iterations, the action of the agent at initialization, and the matrix composed of all channels in the system. Optimize the passive beamforming vector when initializing separately and The step length.

[0064] (2) Algorithm iteration

[0065] In the tth iteration, the training action subnet is based on the current state Generate corresponding actions in Represents the change in the sum of the achievable rates of two consecutive iterations, represents the change of active beamforming vector between two consecutive iterations, H all =[Vec(G1),Vec(G2),Vec(H1),Vec(H2),Vec(H 1,2 )] contains the real form of all CSIs, where H1 = [h 1,1 ,…,h 1,K ] and H2=[h 2,1 ,…,h2,K ], Vec(·) represents the operation of concatenating the real and imaginary parts of the matrix and converting them into a vector. In order to enhance the exploration ability of DRL, the agent selects a random step size with a small preset probability ξ. After getting the action, calculate the current reward

[0066]

[0067] where r con is the reward for algorithm convergence, r iter is the penalty for each iteration of the algorithm. Next, the agent updates the state and uses the experience {s (t) ,a (t) ,r (t) ,s (t+1)} is stored in the playback memory for subsequent training. Then, the algorithm convergence is judged. If the convergence condition is met, that is, and The current active beamforming vector and passive beamforming vector are output.

[0068] Specifically, the proposed model-driven reinforcement learning network structure is as follows Figure 2 The action subnet is composed of three layers of fully connected networks. The "observation" of the intelligent agent is selected as the input of the action subnet. (t) is defined as part of the agent's state, expressed as

[0069]

[0070] The action subnet generates the algorithm step size based on the observation and Then the WMMSE-SCA algorithm is used to update the active and passive beamforming vectors. Specifically, the auxiliary intermediate variables u and λ are first calculated, denoted as

[0071]

[0072] Then, update the active beamforming vector

[0073]

[0074] Next, update the auxiliary intermediate variables u and λ again, and use the step size output by the action subnet The passive beamforming vector of RIS1 is updated and recorded as

[0075]

[0076] in

[0077]

[0078]

[0079] Then update the auxiliary intermediate variables u and λ again, and use the step size output by the action subnet The passive beamforming vector of RIS2 is updated and recorded as

[0080]

[0081] Thus, the agent can get the corresponding action

[0082] (3) Network Update

[0083] Extract a batch of experience s from the experience replay memory (p) ,a (p) ,r (p) ,s (p+1) , where p represents the index of experience. By minimizing the loss function

[0084]

[0085] Update the training review subnet. By minimizing the loss function

[0086]

[0087] Update the training action subnet. Finally, use soft update to update the target action subnet and target evaluation subnet to ensure training stability and convergence, denoted as

[0088]

[0089] The variable τ c and τ a Determines the extent of soft updates.

[0090] The above disclosure is only a preferred embodiment of the present invention, which cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A reconfigurable intelligent surface beamforming method based on model-driven reinforcement learning, characterized in that: Two reconfigurable smart surfaces are introduced to assist the communication process of the system. The method comprises the following steps: (1) Before deployment, randomly initialize the reinforcement learning experience replay memory, training action subnet parameters, target action subnet parameters, training evaluation subnet parameters, and target evaluation subnet parameters; (2) After obtaining the channel estimation result, the passive beamforming vector of the reconfigurable smart surface is randomly initialized; the active beamforming vector is initialized; and the state of the reinforcement learning network is initialized; (3) During iteration, the training action subnet is used to generate corresponding actions based on the current state, calculate rewards and update the agent state; the experience is stored in the replay memory; when the algorithm converges, the active beamforming vector and the passive beamforming vector are output; (4) Extract a batch of experiences from the experience replay memory, update the training action subnet and the training evaluation subnet by minimizing the loss function, and use soft update to update the target action subnet and the target evaluation subnet.

2. The reconfigurable intelligent surface beamforming method based on model-driven reinforcement learning according to claim 1 is characterized in that: Step (2) specifically includes: (2.1) Randomly initialize the passive beamforming vector and (2.2) Calculate the active beamforming vector W using the zero-forcing algorithm (0) =H H (HH H ) -1 , where H = [h1,h2,…,h K ] H is the matrix composed of each user combined channel, K is the number of users, h k is the combined channel of user k; (2.3) Initialize the state of the reinforcement learning network a (0) ,H all They represent the achievable rate and corresponding to the initialization beamforming vector, the change in the achievable rate and between two consecutive iterations, the change in the active beamforming vector between two consecutive iterations, the action of the agent at initialization, and the matrix composed of all channels in the system. Optimize the passive beamforming vector when initializing separately and The step length.

3. The reconfigurable intelligent surface beamforming method based on model-driven reinforcement learning according to claim 2 is characterized in that: Step (3) specifically includes: (3.1) Training the action subnet based on the observation generation algorithm step size and The observation is represented by t represents the time, s (t) [1:6] indicates state s (t) The first 6 elements of ; (3.2) Calculate the auxiliary intermediate variables u and λ. The elements in u and λ are recorded as: P T is the maximum transmit power, w i is the active beamforming vector of the ith user, is the receiving noise variance of user k, u k is the receiving coefficient of user k; (3.3) Update the active beamforming vector according to the auxiliary intermediate variables u and λ calculated in step (3.2); α k is the priority of user k; (3.4) Calculate the auxiliary intermediate variables u and λ again; (3.5) Using the step size output by the action subnet The passive beamforming vector of the first reconfigurable smart surface is updated and recorded as in is the value of the reflection phase φ1 of the first reconfigurable smart surface in the previous iteration, Where G1 and G2 are the channels from AP to the first reconfigurable smart surface and from AP to the second reconfigurable smart surface respectively; H 1,2 is the channel between two reconfigurable smart surfaces; h 1,k 、h 2,k are the channel from the first reconfigurable smart surface to user k and the channel from the second reconfigurable smart surface to user k respectively; represents the real part of the variable, J is the imaginary unit, φ2 is the reflection phase of the first reconfigurable smart surface; (3.6) The third calculation of auxiliary intermediate variables u and λ; (3.7) Using the step size output by the action subnet updating the passive beamforming vector of the second reconfigurable smart surface; (3.8) Calculate the current reward where r con is the reward for algorithm convergence, r iter is the penalty for each iteration of the algorithm, η w They represent the change of the achievable rate and the active beamforming vector in the t-th iteration, the change of the achievable rate and the convergence judgment threshold, and the active beamforming vector convergence judgment threshold respectively; (3.9) Update the agent status in Represents the change in the sum of the achievable rates of two consecutive iterations, represents the change of active beamforming vector between two consecutive iterations, H all =[Vec(G1),Vec(G2),Vec(H1),Vec(H2),Vec(H 1,2 )] contains the real form of all channel matrices, where H1 = [h 1,1 ,…,h 1,K ] and H2=[h 2,1 ,…,h 2,K ],Vec(A)=[R(a 1,1 ,…,a 1,N ,…,a N,1 ,…,a N,N ),I(a 1,1 ,…,a 1,N ,…,a N,1 ,…,a N,N )] represents the operation of concatenating the real and imaginary parts of a matrix and converting them into a vector. i,j Represents the element in the i-th row and j-th column of the matrix; (3.10) Update agent experience {s (t) ,a (t) ,r (t) ,s (t+1) }stored in playback memory; (3.11) satisfies the convergence condition and When , the current active beamforming vector and passive beamforming vector are output.

4. The reconfigurable intelligent surface beamforming method based on model-driven reinforcement learning according to claim 1, characterized in that: Step (4) specifically includes: (4.1) Extract a batch of experience s from the experience replay memory (p) ,a (p) ,r (p) ,s (p+1) ; (4.2) Update the training evaluation subnet by minimizing the loss function (4.3) Update the training action subnet by minimizing the loss function (4.4) Use soft update to update the target action subnet and target evaluation subnet, denoted as

Images