RIS deployment and beam forming joint optimization method based on reinforcement learning

Abstract

The invention discloses an RIS deployment and beamforming joint optimization method based on reinforcement learning, and the method comprises the steps: initializing an experience playback memory, training subnet parameters and target subnet parameters of reinforcement learning before deployment; initializing a passive beamforming vector of the RIS and resetting the position of the RIS at the beginning of transmission each time; performing channel estimation to obtain an initial state of the intelligent agent; in transmission of each frame, estimating a Q value of each action by using a learning action subnet; selecting actions according to the Q value; adjusting an RIS deployment position and a beam forming vector according to the action; estimating a channel and updating the state of the agent; calculating an active beamforming vector, and calculating an agent reward; storing the experience in a playback memory; when the network is trained, experience is extracted from the playback memory, the training action subnet is updated, and when transmission of each frame is finished, soft updating is carried out on the target action subnet. According to the method, the potential of joint design of RIS deployment and beam forming is fully excavated, the performance is good, and the adaptive capacity is high.

Description

Technical Field

[0001] The present invention relates to communication technology, and in particular to a joint optimization method of RIS deployment and beamforming based on reinforcement learning. Background Art

[0002] Wireless networks are constantly evolving. 6G proposes to increase data rates by 10 to 100 times compared to 5G and provide wider and deeper coverage. However, existing technologies face the problems of high hardware costs and high power consumption. Reconfigurable smart surfaces (RIS) have become a promising solution by intelligently manipulating the radio environment. However, in order to fully tap the potential of RIS, the beamforming vectors of the base station (BS) and RIS need to be jointly designed, and the deployment location of RIS also has a significant impact on system performance. However, most of the existing work designs RIS deployment and beamforming vectors separately, and there is still a lack of research on the joint optimization of the two. In recent years, the concept of aerial RIS has been proposed, but most of the existing research has optimized the deployment location of aerial RIS and fixed it, failing to fully utilize the mobility potential of aerial RIS.

[0003] The application of deep learning (DL) technology in physical layer communications has attracted widespread attention in recent years. Existing DL-based research mainly uses neural networks to learn the mapping from channel state information to beamforming vectors. It is necessary to use supervised learning or unsupervised learning to train the network, and it is difficult to achieve real-time adjustment of the deployment position of RIS in the air. The development of deep reinforcement learning (DRL) makes it possible to adjust the deployment position and beamforming of RIS in real time. DRL collects data and trains in the interaction with the environment. However, most existing studies use DRL to directly output the beamforming vector of RIS, facing the problem of large action space and difficulty in network training convergence. It is expected to consider introducing graph neural networks and beamforming design based on codebook to reduce the dimension of action space, improve the convergence speed of network training, alleviate the problem of network training difficulties, achieve joint optimization and improve system performance. Summary of the invention

[0004] Purpose of the invention: To address the problems existing in the prior art, the present invention provides a joint optimization method for RIS deployment and beamforming based on reinforcement learning.

[0005] Technical solution: The RIS deployment and beamforming joint optimization method based on reinforcement learning described in the present invention includes:

[0006] (1) Before the algorithm is deployed, the experience replay memory of reinforcement learning is randomly initialized, and the training subnet parameters and target subnet parameters are initialized;

[0007] (2) Randomly initialize the passive beamforming vector of RIS and reset the position of RIS in the air; estimate the combined channel; initialize the state of the enhanced training subnet;

[0008] (3) During iteration, the training network is used to generate corresponding actions based on the current state, calculate rewards and update the agent state; the experience is stored in the replay memory;

[0009] (4) Extract a batch of experiences from the experience replay memory, update the training subnet by minimizing the loss function, and update the target subnet at the end of each round of training.

[0010] Further, step (2) specifically includes:

[0011] (2.1) Randomly initialize the passive beamforming vector;

[0012] (2.2) Estimation of combined channels

[0013] (2.3) Initialize the state of the agent in Represents the vectorization of the complex matrix H, connecting the real and imaginary parts of its elements into a vector, where and Represent the real and imaginary parts of the variable, H i,j represents the element in the i-th row and j-th column of the channel, K represents the number of users, represents the initial beamforming vector of the lth RIS subarray, and L represents the number of subarrays contained in RIS.

[0014] Furthermore, step (3) specifically includes:

[0015] (3.1) In the tth transmission frame, the Q value is estimated by training the subnet The representative training subnet parameters are When the estimated state s (t) and the Q value of action a, where The superscript (t) indicates the value of the variable in the tth transmission frame; the action a consists of two parts, corresponding to the action space of RIS deployment position adjustment and the action space corresponding to the RIS beamforming adjustment The overall action space is expressed as

[0016] (3.2) Action selection is performed based on the estimated Q values ​​of each action, and the RIS deployment location l R and the beamforming vector φ l Adjust it, record it as

[0017]

[0018] in is the adjustment amount of the RIS position, in the action space corresponding to the adjustment of the RIS deployment position Select from is the adjustment amount of the beamforming vector of the l-th RIS subarray, selected in the action space related to beamforming;

[0019] (3.3) Base station estimates the combined channel Get the state s of the agent (t+1) ; Then use the zero-forcing algorithm to calculate the base station beamforming vector W = H H (HH H ) -1 , and calculate the reward

[0020]

[0021] in is the weighted achievable rate sum of the tth transmission frame, R c >0 is used to adjust the reward range for more stable training, Is the RIS located in the feasible location set Additional penalties when represents the set of possible locations of RIS;

[0022] (3.4)Experience (s (t) ,a (t) ,r (t) ,s (t+1) ) is stored in the experience replay memory for subsequent training; a (t) is the time t from the action space The selected action includes and Two parts; a (t) is the time t from the action space The selected action includes and Two parts.

[0023] Furthermore, for the action space in (3.1), assuming that the aerial RIS moves only in the horizontal space, the action space corresponding to the RIS deployment position adjustment is defined as

[0024]

[0025] in Corresponding to the adjustment amount of the position in different coordinate directions, δ represents the moving step of the RIS in the air between adjacent transmission frames;

[0026] When designing beamforming for RIS, it is assumed that the passive beamforming vectors of RIS must be selected from a predefined codebook, and RIS is further divided into L subarrays, each with N L =N / L reflective elements; the steering vector of RIS is expressed as

[0027]

[0028] Where φ represents the incident angle of the signal and J represents the imaginary unit;

[0029] Define the codeword of the lth subarray is the (l-1)Nth RIS guidance vector L +1 to lN L elements, thus the codebook of the lth subarray is defined as

[0030]

[0031] The passive beamforming vector is expressed as where φ l ∈0,Φ,…,(N-1)Φ represents the direction of the steering vector of the lth subarray; when L=2, the action space for RIS beamforming is expressed as

[0032]

[0033] For L = 4, the action space is defined as

[0034]

[0035] Furthermore, when estimating the Q value using the training subnet in (3.1), the benefits from RIS movement and beamforming vector adjustment are considered to be two independent parts, and the Q values ​​related to these two parts are estimated separately and merged; first, the Q values ​​of the two parts are estimated separately, and then the Q value vectors of the two parts are flattened in different directions to obtain two matrices, and finally the two matrices are added together to obtain the Q value of the entire action space.

[0036] Furthermore, for the structure of the training subnet in (3.1), GNN is used as the basic framework, which includes K user nodes and 1 RIS node; the input of the kth user node is The input of the RIS node is The GNN structure includes input layer, update layer and output layer, as follows:

[0037] Input layer: For user nodes, the input layer is represented as

[0038]

[0039] in is the input of the kth user node, It is a single-layer fully connected neural network with Swish activation function, which maps variables from 2M-dimensional real space to 8M-dimensional real space. Represents a 2M-dimensional real number space;

[0040] For RIS nodes

[0041] φ (1) =f φ,in (φ (0) ),

[0042] where φ (0) is the input of the RIS node, It is a single-layer fully connected neural network with Swish activation function, which maps variables from L-dimensional real space to 16L-dimensional real space;

[0043] Update layer: For user nodes, the update formula for the kth node is

[0044]

[0045] in max(·) represents element-wise maximum pooling; represents the set of user nodes except the kth user node; function and It is a single-layer fully connected neural network with ReLU activation function; the update formula of RIS node is

[0046]

[0047] Where mean(·) represents element-by-element average pooling, K represents the set of user nodes; function and It is also a single-layer fully connected neural network with ReLU activation function;

[0048] Output layer: The output of the user node is

[0049]

[0050] in It is a Dueling structure, which uses two double-layer fully connected networks to estimate the state value function and the state-related action advantage function respectively, and then adds their outputs to obtain the estimated Q value; the output of the RIS node is

[0051] φ (I+2) =f φ,out (φ (I+1) ),

[0052] in It is a Dueling structure that outputs the Q value adjusted for beamforming.

[0053] Further, step (4) specifically includes:

[0054] (4.1) Extract a batch of experience from the experience playback memory Where g represents the index of experience;

[0055] (4.2) Update the training subnet by minimizing the loss function

[0056]

[0057] in Represents the number of extracted experiences, and γ represents the attenuation factor of the reward function;

[0058] (4.3) At the end of each training step, the target subnetwork is updated, expressed as

[0059]

[0060] The variable λ is an auxiliary variable used to stabilize the training process.

[0061] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0062] The present invention innovatively combines the deployment of RIS and beamforming. In the downlink multi-user MISO system assisted by airborne RIS, the achievable rate performance is significantly improved. The proposed reinforcement learning framework does not require the estimation of cascaded channels, avoiding major modifications to the existing system protocol. The present invention proposes a reinforcement training subnet based on a graph neural network, which can better simulate the topological structure of the wireless communication system, greatly reduce the number of network training parameters, and greatly shorten the training time. In general, the present invention provides an airborne RIS-assisted system joint deployment and beamforming optimization solution with practical deployment prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is the combination process of the present invention for the Q values ​​of different partial actions;

[0064] Figure 2 It is a training subnet structure diagram of the present invention;

[0065] Figure 3 It is a Dueling network structure diagram of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be described in detail below with reference to the accompanying drawings and an example of an airborne RIS-assisted downlink multi-user MISO system.

[0067] 1. Channel Model Applicable to This Embodiment

[0068] In a MU-MISO system with one BS and K single-antenna users, an airborne RIS is introduced to assist the communication process. In this embodiment, the number of users is K = 4, and the airborne RIS has N = 64 reflection elements. The maximum transmission power of the BS is P T =20dBm, the carrier frequency is 3GHz, and the distance between adjacent antennas is half a wavelength. The channel from BS to RIS and the channel from RIS to user k are represented by G and h respectively. r,k , the cascade channel is represented by H r,k = The phase shift caused by RIS is expressed as θ = diag(θ), where θ = [θ1, θ2, …, θ N ] T ,θ n represents the phase shift of the nth reflector element of RIS. The signal transmitted by BS to user k is denoted as s k , are assumed to be independent random variables with zero mean and unit variance. The BS uses the beamforming vector w k ∈ processes the signal transmitted to user k, subject to the maximum transmit power constraint The signal received by user k is expressed as

[0069]

[0070] in It represents the noise received by user k, which has a mean of zero and a variance of The signal to noise ratio of the BS to the kth user channel is expressed as

[0071]

[0072] The RIS deployment location and beamforming are jointly designed to maximize the weighted achievable rate and the system. The optimization problem can be expressed as

[0073]

[0074] Where W = [w1,w2,…,w K ], α k represents the priority of the kth user, and the position of RIS is recorded as l R , Represents the set of possible locations of RIS.

[0075] 2. Specific steps of this embodiment

[0076] Reinforcement learning is applied to wireless communication design for deployment and beamforming joint design of downlink multi-user MISO system assisted by airborne RIS. The complete joint beamforming process includes three steps: initialization, action selection, and network update:

[0077] (1) Initialization

[0078] Before the algorithm is deployed, the experience replay memory of reinforcement learning is randomly initialized and the subnet parameters are trained. Target subnet parameters

[0079] During deployment, the passive beamforming vector of the RIS is randomly initialized and the position of the RIS in the air is reset. Then the combined channel is estimated Get the initial state of the agent in

[0080] Concatenate the real and imaginary parts of its elements into a vector where a i,j Represents the element in the i-th row and j-th column.

[0081] (2) Action selection

[0082] In the tth transmission frame, the Q value is estimated by training the subnet in The superscript (t) indicates the value of the variable in the tth transmission frame. Action a consists of two parts, which are the action space corresponding to the adjustment of the RIS deployment position. and the action space corresponding to the RIS beamforming adjustment Therefore, the overall action space can be expressed as

[0083] Then, the action is selected according to the estimated Q value of each action, and the RIS deployment position and beamforming vector are adjusted, which is recorded as

[0084]

[0085] in In the action space adjusted corresponding to the RIS deployment position The action selected in Selection in the action space associated with beamforming.

[0086] After adjusting the RIS deployment position and beamforming vector, the base station estimates the combined channel Get the state s of the agent (t+1) Then use the ZF algorithm to calculate the base station beamforming vector W = HH (HH H ) -1 , and calculate the reward

[0087]

[0088] in is the weighted achievable rate sum of the tth transmission frame, R c >0 is used to adjust the reward range for more stable training, Is the RIS located in the feasible location set Additional penalties for other situations.

[0089] Finally, experience (t) ,a (t) ,r (t) ,s (t+1) ) is stored in the experience replay memory for subsequent training.

[0090] Specifically, for the action space, assuming that the aerial RIS moves only in the horizontal space, the action space corresponding to the RIS deployment position adjustment can be defined as

[0091]

[0092] Where δ represents the moving step of the air RIS between adjacent transmission frames.

[0093] When designing beamforming for RIS, it is assumed that the passive beamforming vectors of RIS must be selected from a predefined codebook, and RIS is further divided into L subarrays, each with N L =N / L reflective elements. The steering vector of RIS is expressed as

[0094]

[0095] Define the codeword of the lth subarray is the (l-1)Nth RIS guidance vector L +1 to lN L elements, thus the codebook of the lth subarray is defined as

[0096]

[0097] The passive beamforming vector can be expressed as where φ l ∈0,Φ,…,(N-1)Φ represents the direction of the l-th subarray steering vector. When L=2, the action space for RIS beamforming can be expressed as

[0098]

[0099] For L = 4, the action space can be similarly defined as

[0100]

[0101] When using the training subnet to estimate the Q value, the benefits from RIS movement and beamforming vector adjustment are considered to be two independent parts, and the Q values ​​related to these two parts are estimated separately and combined. Figure 1 The merging method shown in FIG. 1 first estimates the Q values ​​of the two parts separately, then flattens the Q value vectors of the two parts in different directions to obtain two matrices, and finally adds the two matrices to obtain the Q value of the entire action space.

[0102] The structure of the training subnet is as follows Figure 2 As shown in Figure 1, GNN is used as the basic framework, which includes K user nodes and 1 RIS node. The input of the kth user node is The input of the RIS node is The GNN structure includes input layer, update layer and output layer, as follows:

[0103] Input layer: For user nodes, the input layer can be expressed as

[0104]

[0105] in is a single-layer fully connected neural network with a Swish activation function. Similarly, for the RIS node

[0106] φ (1) =f φ,in (φ (0) ),

[0107] in It is a single-layer fully connected neural network with Swish activation function.

[0108] Update layer: For user nodes, the update formula for the kth node is

[0109]

[0110] in max(·) represents element-wise max pooling. Represents the set of user nodes except the kth user node. Function and It is a single-layer fully connected neural network with ReLU activation function. The update formula of RIS node is

[0111]

[0112] Where mean(·) represents element-by-element average pooling, and f represents the set of user nodes. Function and It is also a single-layer fully connected neural network with ReLU activation function.

[0113] Output layer: The output of the user node is

[0114]

[0115] in For Figure 3 The Dueling structure shown in Figure 1 uses two double-layer fully connected networks to estimate the state value function and the state-related action advantage function respectively, and then adds their outputs to obtain the estimated Q value. The output of the RIS node is

[0116] φ (I+2) =f φ,out (φ (I+1) ),

[0117] in It is a Dueling structure that outputs the Q value adjusted for beamforming.

[0118] (3) Network Update

[0119] Extract a batch of experience from the experience replay memory where g represents the index of experience. By minimizing the loss function

[0120]

[0121] Update the training subnet, where represents the number of extracted experiences, and γ represents the attenuation factor of the reward function. At the end of each training step, the target subnetwork is updated, which can be expressed as

[0122]

[0123] The variable λ is an auxiliary variable used to stabilize the training process.

[0124] The above disclosure is only a preferred embodiment of the present invention, which cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A joint optimization method for RIS deployment and beamforming based on reinforcement learning, characterized in that: The method comprises the following steps: (1) Before the algorithm is deployed, the experience replay memory of reinforcement learning is randomly initialized, and the training subnet parameters and target subnet parameters are initialized; (2) Randomly initialize the passive beamforming vector of RIS and reset the position of RIS in the air; estimate the combined channel; initialize the state of the enhanced training subnet; (3) During iteration, the training network is used to generate corresponding actions based on the current state, calculate rewards and update the agent state; the experience is stored in the replay memory; (4) Extract a batch of experiences from the experience replay memory, update the training subnet by minimizing the loss function, and update the target subnet at the end of each round of training.

2. The method for joint optimization of RIS deployment and beamforming based on reinforcement learning according to claim 1, characterized in that: Step (2) specifically includes: (2.1) Randomly initialize the passive beamforming vector; (2.2) Estimation of combined channel (2.3) Initialize the state of the agent in Represents the vectorization of the complex matrix H, connecting the real and imaginary parts of its elements into a vector, where and Represent the real and imaginary parts of the variable, H i,j represents the element in the i-th row and j-th column of the channel, K represents the number of users, represents the initial beamforming vector of the lth RIS subarray, and L represents the number of subarrays contained in RIS.

3. The method for joint optimization of RIS deployment and beamforming based on reinforcement learning according to claim 1, characterized in that: Step (3) specifically includes: (3.1) In the tth transmission frame, the Q value is estimated by training the subnet The representative training subnet parameters are When the estimated state s (t) and the Q value of action a, where The superscript (t) indicates the value of the variable in the tth transmission frame; the action a consists of two parts, corresponding to the action space of RIS deployment position adjustment and the action space corresponding to the RIS beamforming adjustment The overall action space is expressed as (3.2) Action selection is performed based on the estimated Q values ​​of each action, and the RIS deployment location is adjusted. and the beamforming vector φ l Adjust it, record it as in is the adjustment amount of the RIS position, in the action space corresponding to the adjustment of the RIS deployment position Select from is the adjustment amount of the beamforming vector of the l-th RIS subarray, selected in the action space related to beamforming; (3.3) Base station estimates the combined channel Get the state s of the agent (t+1) ; Then use the zero-forcing algorithm to calculate the base station beamforming vector W = H H (HH H ) -1 , and calculate the reward in is the weighted achievable rate sum of the tth transmission frame, R c >0 is used to adjust the reward range for more stable training, Is the RIS located in the feasible location set Additional penalties when represents the possible location set of RIS; (3.4) The experience (s (t) ,a (t) ,r (t) ,s (t+1) ) is stored in the experience replay memory for subsequent training; a (t) is the time from the action space The selected action includes and Two parts.

4. The method for joint optimization of RIS deployment and beamforming based on reinforcement learning according to claim 3 is characterized in that: The action space of step (3.1) specifically includes: Assuming that the aerial RIS moves only in the horizontal space, the action space corresponding to the RIS deployment position adjustment is defined as in Corresponding to the adjustment amount of the position in different coordinate directions, δ represents the moving step of the RIS in the air between adjacent transmission frames; When designing beamforming for RIS, it is assumed that the passive beamforming vectors of RIS must be selected from a predefined codebook, and RIS is further divided into L subarrays, each with N L =N / L reflective elements; the steering vector of RIS is expressed as Where φ represents the incident angle of the signal and J represents the imaginary unit; Define the codeword of the lth subarray is the (l-1)Nth RIS guidance vector L +1 to lN L elements, thus the codebook of the lth subarray is defined as The passive beamforming vector is expressed as where φ l ∈0,φ,…,(N-1)φ represents the direction of the l-th subarray steering vector; when L=2, the action space for RIS beamforming is expressed as For L = 4, the action space is defined as 5. The method for joint optimization of RIS deployment and beamforming based on reinforcement learning according to claim 3, characterized in that: The estimated Q value of step (3.1) specifically includes: When using the training subnet to estimate the Q value, the benefits from RIS movement and beamforming vector adjustment are considered to be two independent parts, and the Q values ​​related to these two parts are estimated separately and merged; first, the Q values ​​of the two parts are estimated separately, and then the Q value vectors of the two parts are flattened in different directions to obtain two matrices, and finally the two matrices are added together to obtain the Q value of the entire action space.

6. The method for joint optimization of RIS deployment and beamforming based on reinforcement learning according to claim 3, characterized in that: The network architecture of the subnet in step (3.1) specifically includes: The structure of the training subnet uses GNN as the basic framework, which includes K user nodes and 1 RIS node; the input of the kth user node is The input of the RIS node is The GNN structure includes input layer, update layer and output layer, as follows: Input layer: For user nodes, the input layer is represented as in is the input of the kth user node, f u,in : It is a single-layer fully connected neural network with Swish activation function, which maps variables from 2M-dimensional real space to 8M-dimensional real space. Represents a 2M-dimensional real number space; For RIS nodes f (1) =f φ,in (f ( 0)), where φ (0) is the input of the RIS node, f φ,in : It is a single-layer fully connected neural network with Swish activation function, which maps variables from L-dimensional real space to 16L-dimensional real space; Update layer: For user nodes, the update formula for the kth node is in max(·) represents element-wise maximum pooling; represents the set of user nodes except the kth user node; function and It is a single-layer fully connected neural network with ReLU activation function; the update formula of RIS node is Where mean(·) represents element-wise average pooling, Represents a user node set; function and It is also a single-layer fully connected neural network with ReLU activation function; Output layer: The output of the user node is where f u,out : It is a Dueling structure, which uses two double-layer fully connected networks to estimate the state value function and the state-related action advantage function respectively, and then adds their outputs to obtain the estimated Q value; the output of the RIS node is f (l+2) =f φ,out (f (I+1) ), where f φ,out : It is a Dueling structure that outputs the Q value adjusted for beamforming.

7. The method for joint optimization of RIS deployment and beamforming based on reinforcement learning according to claim 3, characterized in that: Step (4) specifically includes: (4.1) Extract a batch of experience from the experience playback memory Where g represents the index of experience; (4.2) Update the training subnet by minimizing the loss function in Represents the number of extracted experiences, and γ represents the attenuation factor of the reward function; (4.3) At the end of each training step, the target subnetwork is updated, expressed as The variable λ is an auxiliary variable used to stabilize the training process.

Images