A joint optimization method for RIS deployment and beamforming based on reinforcement learning

Abstract

The present invention discloses a joint optimization method for RIS deployment and beamforming based on reinforcement learning, comprising: before deployment, initializing the reinforcement learning experience replay memory, training subnet parameters, and target subnet parameters; at the beginning of each transmission, initializing the passive beamforming vector of the RIS and resetting the position of the RIS; performing channel estimation to obtain the initial state of the agent; during each frame transmission, using the learning action subnet to estimate the Q value of each action; selecting an action based on the Q value; adjusting the RIS deployment position and beamforming vector based on the action; estimating the channel and updating the agent state; calculating the active beamforming vector and calculating the agent's reward; storing the experience in the replay memory; extracting the experience from the replay memory during network training, updating the training action subnet, and performing a soft update on the target action subnet at the end of each frame transmission. The present invention fully taps the potential of joint design of RIS deployment and beamforming, with good performance and strong adaptability.

Description

Technical Field

[0001] The present invention relates to communication technology, and in particular to a reinforcement learning-based joint optimization method for RIS deployment and beamforming. Background Art

[0002] Wireless networks are constantly evolving. 6G promises to increase data rates 10 to 100 times compared to 5G, while also providing wider and deeper coverage. However, existing technologies face challenges with high hardware costs and power consumption. Reconfigurable smart surfaces (RIS) offer a promising solution by intelligently manipulating the radio environment. However, to fully exploit the potential of RIS, the beamforming vectors of the base station (BS) and RIS need to be jointly designed. Furthermore, the RIS deployment location significantly impacts system performance. While most existing work has focused on designing RIS deployment and beamforming vectors separately, research on jointly optimizing both remains relatively lacking. The concept of aerial RIS has been proposed in recent years, but most existing research focuses on optimizing the deployment location of the aerial RIS and then fixing it, failing to fully exploit its mobility potential.

[0003] The application of deep learning (DL) technology in physical layer communications has attracted widespread attention in recent years. Existing DL-based research primarily uses neural networks to learn the mapping from channel state information to beamforming vectors. This requires supervised or unsupervised learning to train the network, making it difficult to achieve real-time adjustment of the RIS deployment position in the air. The development of deep reinforcement learning (DRL) has made it possible to adjust the RIS deployment position and beamforming in real time. DRL collects data and trains through interaction with the environment. However, most existing research uses DRL to directly output the RIS beamforming vectors, which faces the problem of a large action space and difficulty in network training convergence. It is expected that the introduction of graph neural networks and codebook-based beamforming design will reduce the dimensionality of the action space, improve the convergence speed of network training, alleviate the network training difficulties, achieve joint optimization, and improve system performance. Summary of the Invention

[0004] Purpose of the invention: To address the problems existing in the prior art, the present invention provides a joint optimization method for RIS deployment and beamforming based on reinforcement learning.

[0005] Technical Solution: The reinforcement learning-based RIS deployment and beamforming joint optimization method of the present invention includes:

[0006] (1) Before algorithm deployment, randomly initialize the experience replay memory of reinforcement learning, training subnet parameters, and target subnet parameters;

[0007] (2) Randomly initialize the passive beamforming vector of the RIS and reset the position of the RIS in the air; estimate the combined channel; initialize the state of the reinforcement training subnet;

[0008] (3) During iteration, the training network is used to generate corresponding actions based on the current state, calculate rewards and update the agent state; the experience is stored in the replay memory;

[0009] (4) Extract a batch of experiences from the experience replay memory, update the training subnet by minimizing the loss function, and update the target subnet at the end of each round of training.

[0010] Furthermore, step (2) specifically includes:

[0011] (2.1) Randomly initialize the passive beamforming vector;

[0012] (2.2) Estimation of the combined channel

[0013] (2.3) Initialize the state of the agent in Represents the vectorization of the complex matrix H, connecting the real and imaginary parts of its elements into a vector, where and Represent the real and imaginary parts of the variable, H i,j represents the element in the i-th row and j-th column of the channel, K represents the number of users, represents the initial beamforming vector of the lth RIS subarray, and L represents the number of subarrays contained in RIS.

[0014] Furthermore, step (3) specifically includes:

[0015] (3.1) In the tth transmission frame, the Q value is estimated by training the subnet The representative training subnet parameters are When the estimated state s (t) and the Q value of action a, where The superscript (t) indicates the value of the variable in the tth transmission frame; the action a consists of two parts, corresponding to the action space of RIS deployment position adjustment and the action space corresponding to RIS beamforming adjustment The overall action space is expressed as

[0016] (3.2) Action selection is performed based on the estimated Q value of each action, and the RIS deployment position l R and the beamforming vector φ l Adjust, record as

[0017]

[0018] in is the adjustment amount of RIS position, in the action space corresponding to the adjustment of RIS deployment position Select from is the adjustment amount of the beamforming vector of the l-th RIS subarray, selected in the beamforming-related action space;

[0019] (3.3) Base station estimates combined channel Get the state s of the agent (t+1) ; Then use the zero-forcing algorithm to calculate the base station beamforming vector W = H H (HH H ) -1 , and calculate the reward

[0020]

[0021] in is the weighted achievable rate sum of the t-th transmission frame, R c >0 is used to adjust the reward range for more stable training, Is the RIS located in the feasible location set Additional penalties when represents the set of possible locations of RIS;

[0022] (3.4)Experience (s (t) ,a (t) ,r (t) ,s (t+1) ) is stored in the experience replay memory for subsequent training; a (t) is the time from the action space The selected action includes and Two parts; a (t) is the time from the action space The selected action includes and Two parts.

[0023] Furthermore, for the action space in (3.1), assuming that the aerial RIS moves only in the horizontal space, the action space corresponding to the RIS deployment position adjustment is defined as

[0024]

[0025] in Corresponding to the adjustment amount of the position in different coordinate directions, δ represents the moving step of the RIS in the air between adjacent transmission frames;

[0026] When designing beamforming for RIS, it is assumed that the passive beamforming vectors of RIS must be selected from a predefined codebook, and RIS is further divided into L sub-arrays, each with N L =N / L reflective elements; the steering vector of RIS is expressed as

[0027]

[0028] Where φ represents the incident angle of the signal and J represents the imaginary unit;

[0029] Define the codeword of the lth subarray is the (l-1)Nth RIS guidance vector L +1 to lN L elements, thus the codebook of the lth subarray is defined as

[0030]

[0031] The passive beamforming vector is expressed as where φ l ∈0,Φ,…,(N-1)Φ represents the direction of the steering vector of the lth subarray; when L=2, the action space for RIS beamforming is expressed as

[0032]

[0033] For L=4, the action space is defined as

[0034]

[0035] Furthermore, when estimating the Q value using the training subnet in (3.1), the benefits from the RIS movement and the beamforming vector adjustment are considered to be two independent parts, and the Q values related to these two parts are estimated separately and merged. First, the Q values of the two parts are estimated separately, and then the Q value vectors of the two parts are flattened in different directions to obtain two matrices. Finally, these two matrices are added together to obtain the Q value of the entire action space.

[0036] Furthermore, for the structure of the training subnet in (3.1), GNN is used as the basic framework, which includes K user nodes and 1 RIS node; the input of the kth user node is The input of the RIS node is The GNN structure includes input layer, update layer and output layer, as follows:

[0037] Input layer: For user nodes, the input layer is represented as

[0038]

[0039] in is the input of the kth user node, It is a single-layer fully connected neural network with Swish activation function, which maps variables from 2M-dimensional real space to 8M-dimensional real space. Represents a 2M-dimensional real number space;

[0040] For RIS nodes

[0041] φ (1) =f φ,in (φ (0) ),

[0042] where φ (0) is the input of the RIS node, It is a single-layer fully connected neural network with Swish activation function, which maps variables from L-dimensional real space to 16L-dimensional real space;

[0043] Update layer: For the user node, the update formula of the kth node is

[0044]

[0045] in max(·) represents element-wise maximum pooling; Represents the set of user nodes except the kth user node; function and It is a single-layer fully connected neural network with ReLU activation function; the update formula of RIS node is

[0046]

[0047] Where mean(·) represents element-wise average pooling, K represents the set of user nodes; function and It is also a single-layer fully connected neural network with ReLU activation function;

[0048] Output layer: The output of the user node is

[0049]

[0050] in Dueling structure, two double-layer fully connected networks are used to estimate the state value function and the state-related action advantage function respectively, and then their outputs are added together to obtain the estimated Q value; the output of the RIS node is

[0051] φ (I+2) =f φ,out (φ (I+1) ),

[0052] in It is a Dueling structure that outputs the Q value adjusted for beamforming.

[0053] Furthermore, step (4) specifically includes:

[0054] (4.1) Extract a batch of experience from the experience replay memory Where g represents the index of experience;

[0055] (4.2) Update the training subnet by minimizing the loss function

[0056]

[0057] in Represents the number of extracted experiences, and γ represents the attenuation factor of the reward function;

[0058] (4.3) At the end of each training step, the target subnetwork is updated and expressed as

[0059]

[0060] The variable λ is an auxiliary variable used to stabilize the training process.

[0061] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0062] This invention innovatively combines RIS deployment and beamforming, significantly improving achievable rate performance in an airborne RIS-assisted downlink multi-user MISO system. The proposed reinforcement learning framework eliminates the need for cascaded channel estimation, avoiding significant modifications to existing system protocols. Furthermore, the invention proposes a graph neural network-based reinforcement training subnet that better simulates the topology of wireless communication systems, significantly reducing the number of network training parameters and training time. Overall, this invention provides a promising solution for the combined deployment and beamforming optimization of airborne RIS-assisted systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is the process of combining the Q values of different partial actions in the present invention;

[0064] Figure 2 It is a structural diagram of the training subnet of the present invention;

[0065] Figure 3 This is the Dueling network structure diagram of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be described in detail below with reference to the accompanying drawings and an example of an airborne RIS-assisted downlink multi-user MISO system.

[0067] 1. Channel Model Applicable to This Embodiment

[0068] In a MU-MISO system with one BS and K single-antenna users, an airborne RIS is introduced to assist the communication process. In this embodiment, the number of users is K = 4, and the airborne RIS has N = 64 reflection elements. The maximum transmission power of the BS is P T = 20dBm, the carrier frequency is 3GHz, and the distance between adjacent antennas is half a wavelength. The channel from BS to RIS and the channel from RIS to user k are represented by G and h respectively. r,k , the cascade channel is represented by H r,k = The phase shift caused by RIS is expressed as Θ = diag(θ), where θ = [θ1, θ2, …, θ N ] T ,θ n The signal transmitted by BS to user k is denoted as s k , are assumed to be independent random variables with zero mean and unit variance. The BS uses the beamforming vector w k ∈ processes the signal transmitted to user k, subject to the maximum transmit power constraint The signal received by user k is expressed as

[0069]

[0070] in It represents the noise received by user k, which has a mean of zero and a variance of The signal to interference noise ratio of the BS to the kth user channel is expressed as

[0071]

[0072] The RIS deployment location and beamforming are jointly designed to maximize the weighted achievable rate of the system. The optimization problem can be expressed as

[0073]

[0074] Where W=[w1,w2,…,w K ], α k represents the priority of the kth user, and the position of RIS is recorded as l R , Represents the set of possible locations of RIS.

[0075] 2. Specific steps of this embodiment

[0076] Reinforcement learning is applied to wireless communication design for deployment and joint beamforming design of downlink multi-user MISO systems assisted by airborne RIS. The complete joint beamforming process includes three steps: initialization, action selection, and network update:

[0077] (1) Initialization

[0078] Before the algorithm is deployed, the experience replay memory of reinforcement learning is randomly initialized and the subnet parameters are trained. Target subnet parameters

[0079] During the deployment process, the passive beamforming vector of the RIS is randomly initialized and the position of the RIS in the air is reset. Then the combined channel is estimated Get the initial state of the agent in

[0080] Concatenate the real and imaginary parts of its elements into a vector where a i,j Represents the element in row i and column j.

[0081] (2) Action selection

[0082] In the tth transmission frame, the Q value is estimated by training the subnet in The superscript (t) indicates the value of the variable in the tth transmission frame. Action a consists of two parts, which are the action space corresponding to the RIS deployment position adjustment. and the action space corresponding to RIS beamforming adjustment Therefore, the overall action space can be expressed as

[0083] Then, the action is selected based on the estimated Q value of each action, and the RIS deployment position and beamforming vector are adjusted, which is recorded as

[0084]

[0085] in In the action space adjusted corresponding to the RIS deployment position The action selected in Selection in the action space associated with beamforming.

[0086] After adjusting the RIS deployment position and beamforming vector, the base station estimates the combined channel Get the state s of the agent (t+1) Then use the ZF algorithm to calculate the base station beamforming vector W = HH (HH H ) -1 , and calculate the reward

[0087]

[0088] in is the weighted achievable rate sum of the t-th transmission frame, R c >0 is used to adjust the reward range for more stable training, Is the RIS located in the feasible location set Additional penalties for other situations.

[0089] Finally, experience (s (t) ,a (t) ,r (t) ,s (t+1) ) is stored in the experience replay memory for subsequent training.

[0090] Specifically, for the action space, assuming that the aerial RIS moves only in the horizontal space, the action space corresponding to the RIS deployment position adjustment can be defined as

[0091]

[0092] Where δ represents the moving step of the air RIS between adjacent transmission frames.

[0093] When designing beamforming for RIS, it is assumed that the passive beamforming vectors of RIS must be selected from a predefined codebook, and RIS is further divided into L sub-arrays, each with N L =N / L reflective elements. The steering vector of RIS is expressed as

[0094]

[0095] Define the codeword of the lth subarray is the (l-1)Nth RIS guidance vector L +1 to lN L elements, thus the codebook of the lth subarray is defined as

[0096]

[0097] The passive beamforming vector can be expressed as where φ l ∈0,Φ,…,(N-1)Φ represents the direction of the l-th subarray steering vector. When L=2, the action space for RIS beamforming can be expressed as

[0098]

[0099] For L = 4, the action space can be similarly defined as

[0100]

[0101] When using the training subnet to estimate the Q value, the benefits from RIS movement and beamforming vector adjustment are considered to be two independent parts, and the Q values related to these two parts are estimated separately and combined. Figure 1 The merging method shown in FIG1 first estimates the Q value of the two parts separately, then flattens the Q value vectors of the two parts in different directions to obtain two matrices, and finally adds the two matrices to obtain the Q value of the entire action space.

[0102] The structure of the training subnet is as follows Figure 2 As shown in Figure 2, GNN is used as the basic framework, which includes K user nodes and 1 RIS node. The input of the kth user node is The input of the RIS node is The GNN structure includes input layer, update layer and output layer, as follows:

[0103] Input layer: For user nodes, the input layer can be expressed as

[0104]

[0105] in is a single-layer fully connected neural network with Swish activation function. Similarly, for RIS node

[0106] φ (1) =f φ,in (φ (0) ),

[0107] in It is a single-layer fully connected neural network with Swish activation function.

[0108] Update layer: For the user node, the update formula of the kth node is

[0109]

[0110] in max(·) represents element-wise max pooling. Represents the set of user nodes except the k-th user node. Function and It is a single-layer fully connected neural network with ReLU activation function. The update formula of RIS node is

[0111]

[0112] Where mean(·) represents element-wise average pooling, and f represents the set of user nodes. Function and It is also a single-layer fully connected neural network with ReLU activation function.

[0113] Output layer: The output of the user node is

[0114]

[0115] in For example Figure 3 The Dueling structure shown in Figure 1 uses two double-layer fully connected networks to estimate the state value function and the state-related action advantage function respectively, and then adds their outputs to obtain the estimated Q value. The output of the RIS node is

[0116] φ (I+2) =f φ,out (φ (I+1) ),

[0117] in It is a Dueling structure that outputs the Q value adjusted for beamforming.

[0118] (3) Network Update

[0119] Extract a batch of experience from the experience replay memory Where g represents the index of experience. By minimizing the loss function

[0120]

[0121] Update the training subnet, where Represents the number of extracted experiences, and γ represents the attenuation factor of the reward function. At the end of each training step, the target subnetwork is updated, which can be expressed as

[0122]

[0123] The variable λ is an auxiliary variable used to stabilize the training process.

[0124] The above disclosure is only a preferred embodiment of the present invention and cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A joint optimization method for RIS deployment and beamforming based on reinforcement learning, characterized in that: The method comprises the following steps: (1) Randomly initialize the experience replay memory, training subnet parameters, and target subnet parameters of reinforcement learning; (2) Randomly initialize the passive beamforming vector of the RIS and reset the position of the RIS in the air; estimate the combined channel; initialize the state of the training subnet; (3) During iteration, the training subnet is used to generate corresponding actions based on the current state, calculate rewards and update the agent state; and store the experience in the experience replay memory; (4) Extract a batch of experiences from the experience replay memory, update the training subnet by minimizing the loss function, and update the target subnet at the end of each round of training; Step (3) specifically includes: (3.1) At the transmission frame at time t, the Q value estimated by the training subnet is Q π (s (t) ,a;θ train ), Q π (s (t) ,a;θ train ) represents the training subnet parameter θ train When the estimated state s (t) and the Q value of action a, where The superscript (t) indicates the value of the variable in the transmission frame at time t. represents the combined channel of the K-th user node, They represent the beamforming vectors of the 1st, 2nd, ...Lth RIS subarrays in the transmission frame at time t; action a consists of two parts, corresponding to the action space of RIS deployment position adjustment. and RIS beamforming adjustment action space The overall action space is expressed as (3.2) Action selection is performed based on the estimated Q value of each action, and the RIS deployment position l R and the beamforming vector φ l Adjust, record as in is the adjustment amount of RIS position, in the action space corresponding to the adjustment of RIS deployment position Select from is the adjustment amount of the l-th RIS subarray beamforming vector, selected in the action space corresponding to the RIS beamforming adjustment; (3.3) Base station estimates combined channel Get the state s of the agent (t+1) ; Then use the zero-forcing algorithm to calculate the base station beamforming vector W = H H (HH H ) -1 , and calculate the reward in is the weighted achievable rate sum of the transmission frames at time t, R c >0 is used to adjust the reward range for more stable training, Is the RIS located in the feasible location set Additional penalties when represents the set of possible locations of RIS; (3.4)Experience (s (t) ,a (t) ,r (t) ,s (t+1) ) is stored in the experience replay memory for subsequent training; a (t) is the time from the action space The selected action includes and Two parts; The network architecture of the training subnet in step (3.1) specifically includes: The structure of the training subnet uses GNN as the basic framework, which includes K user nodes and 1 RIS node; the input of the kth user node is and Represent the real and imaginary parts of the variable respectively. The input of the RIS node is The GNN structure includes input layer, update layer and output layer, as follows: Input layer: For user nodes, the input layer is represented as in It is a single-layer fully connected neural network with Swish activation function, which maps variables from 2M-dimensional real space to 8M-dimensional real space. Represents a 2M-dimensional real number space; For RIS nodes f (1) =f φ,in (f (0) ), in It is a single-layer fully connected neural network with Swish activation function, which maps variables from L-dimensional real space to 16L-dimensional real space; Update layer: For the user node, the update formula of the kth node is in max(·) represents element-wise maximum pooling; Represents the set of user nodes except the kth user node; function and It is a single-layer fully connected neural network with ReLU activation function; the update formula of RIS node is Where mean(·) represents element-wise average pooling, Represents a set of user nodes; function and It is also a single-layer fully connected neural network with ReLU activation function; Output layer: The output of the user node is in Dueling structure, two double-layer fully connected networks are used to estimate the state value function and the state-related action advantage function respectively, and then their outputs are added together to obtain the estimated Q value; the output of the RIS node is φ (I+2) =f φ,out (φ (I+1) ), in It is a Dueling structure that outputs the Q value adjusted for beamforming.

2. The reinforcement learning-based RIS deployment and beamforming joint optimization method according to claim 1, characterized in that: Step (2) specifically includes: (2.1) Randomly initialize the passive beamforming vector; (2.2) Estimation of combined channels (2.3) Initialize the state of the agent in Indicates the vectorization of the complex matrix H, connecting the real and imaginary parts of its elements into a vector, H i,j represents the element in the i-th row and j-th column of the channel, represents the initial beamforming vector of the lth RIS subarray, and L represents the number of subarrays contained in RIS.

3. The reinforcement learning-based RIS deployment and beamforming joint optimization method according to claim 1, characterized in that: The action space of step (3.1) specifically includes: Assuming that the aerial RIS moves only in the horizontal space, the action space corresponding to the RIS deployment position adjustment is defined as in Corresponding to the adjustment amount of the position in different coordinate directions, δ represents the moving step of the RIS in the air between adjacent transmission frames; When designing beamforming for RIS, it is assumed that the passive beamforming vectors of RIS must be selected from a predefined codebook, and RIS is further divided into L sub-arrays, each with N L =N / L reflective elements; the steering vector of RIS is expressed as Where φ represents the incident angle of the signal and j represents the imaginary unit; Define the codeword of the lth subarray is the (l-1)Nth RIS guidance vector L +1 to lN L elements, so the codebook of the lth subarray is defined as The passive beamforming vector is expressed as where φ l ∈0,Φ,…,(N-1)Φ represents the direction of the steering vector of the lth subarray; when L=2, the action space for RIS beamforming is expressed as For L=4, the action space is defined as 4. The reinforcement learning-based RIS deployment and beamforming joint optimization method according to claim 1, characterized in that: The estimated Q value of step (3.1) specifically includes: When using the training subnet to estimate Q values, the benefits from RIS movement and beamforming vector adjustment are considered to be two independent parts. The Q values related to these two parts are estimated separately and then merged. First, the Q values of the two parts are estimated separately. Then, the Q value vectors of the two parts are flattened in different directions to obtain two matrices. Finally, these two matrices are added together to obtain the Q value of the entire action space.

5. The reinforcement learning-based RIS deployment and beamforming joint optimization method according to claim 1, characterized in that: Step (4) specifically includes: (4.1) Extract a batch of experience from the experience replay memory in An index that represents experience; (4.2) Update the training subnet by minimizing the loss function in represents the number of extracted experiences, γ represents the attenuation factor of the reward function, and θ tar Represents the target subnet; (4.3) At the end of each training step, update the target subnet θ tar , expressed as i tar =λθ train +(1-λ)θ tar The variable λ is an auxiliary variable used to stabilize the training process.

Images