A RIS-assisted MU-MISO communication system intelligent beamforming method based on deep reinforcement learning

Abstract

The present invention discloses an intelligent beamforming design method for a MU-MISO communication system assisted by RIS, proposes a CSI-based DDPG model for jointly optimizing the active transmit beamforming matrix of the base station and the reflection coefficient vector of RIS, and uses reinforcement learning technology to adaptively optimize the beamforming parameters of the communication system through interaction with the environment, thereby achieving optimal transmission performance under actual channel conditions. The algorithm proposed in the present invention exhibits good robustness and convergence under different transmit powers and numbers of RIS reflection units, significantly improving the transmission rate of the MU-MISO communication system. In addition, the algorithm can effectively cope with rapidly changing channel environments, providing important support for the application of RIS in actual wireless communication systems. By combining reinforcement learning with beamforming technology, the present invention proposes an efficient joint optimization method, which provides new ideas for the development of future 6G communication systems and has important application prospects and practical value.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to an intelligent beamforming design method for a multi-user multiple input single output (MU-MISO) communication system assisted by a reconfigurable intelligent surface (RIS), which is suitable for communication scenarios with high real-time requirements, large amounts of information, and high security requirements, such as financial data transmission, military communications, etc. Background Art

[0002] Future 6G wireless communication systems require higher system throughput, spectrum efficiency, and energy efficiency. However, existing massive multiple input multiple output (MIMO) systems and millimeter wave technology face many challenges in practical applications. Massive MIMO systems rely on a large number of antennas and complex RF links, resulting in high hardware costs and huge energy consumption. In addition, channel estimation difficulties and multi-user interference limit their actual performance. Although millimeter wave technology provides high data rates, its signals are susceptible to path loss, building obstruction, and weather conditions, resulting in unstable transmission and increasing system complexity and cost.

[0003] Traditional communication research believes that the wireless channel between transceivers is random and uncontrollable. Therefore, the physical layer security transmission method is limited to establishing a channel model by accurately measuring the channel and passively adapting to the wireless channel environment, but cannot change the transmission environment. Due to the randomness of the wireless channel, it is difficult to ensure the stable establishment of the dominant channel, resulting in random fluctuations in the security transmission performance. This limits the application effect of physical layer security technology to a certain extent. RIS, as a new type of adjustable metamaterial device, consists of a large number of passive and configurable reflection units. By adjusting the phase of the incident signal, the reflected signal and the signal on other paths can be mutually enhanced or offset, achieving signal enhancement / weakening in a specific direction, and achieving the purpose of reconstructing the wireless channel environment. RIS technology can effectively improve the wireless channel propagation environment, improve the system transmission efficiency and coverage, and has the characteristics of low power consumption, low cost and easy deployment. Therefore, RIS can not only be used as a green communication technology to improve the security of physical layer communications, but also provide new degrees of freedom for the design of future wireless communication systems. While ensuring energy efficiency, it can achieve higher data transmission rates and adaptively reconstruct the channel transmission environment, opening up new ways for secure transmission at the physical layer.

[0004] In order to fully utilize the advantages of RIS, the joint design of active beamforming of base stations and passive beamforming of RIS must be considered comprehensively. In general, it is very difficult to mathematically and statistically analyze the synthetic random channel model of RIS reflection link and direct link, and the statistical analysis and derivation of synthetic random channels are still in the initial research stage. With the high computational complexity brought by RIS phase shift control, the optimization of multi-antenna and multi-user secure beamforming becomes very difficult. Existing schemes usually use multiple iterations of convex optimization methods to obtain the optimal solutions of RIS phase shift matrix and beamforming covariance matrix respectively. The complexity of the optimization algorithm increases in high order with the increase of the number of antennas, reflection units and users. Traditional optimization algorithms have high computational complexity and long solution time when solving the above problems, especially in multi-user, multi-antenna and high-frequency band communication environments. Emerging computational methods such as reinforcement learning and deep learning provide new ideas for optimizing the base station active transmit beamforming matrix and RIS reflection coefficient vector.

[0005] The present invention proposes a deep deterministic policy gradient (DDPG) method based on channel state information (CSI). Through interaction and learning with the environment, the beamforming parameters of the communication system are gradually optimized to achieve optimal transmission performance under actual channel conditions, significantly improve the transmission rate and coverage of the MU-MISO communication system, and provide strong technical support for the development of 6G wireless communication systems. Summary of the invention

[0006] The purpose of the present invention is to provide a RIS-assisted MU-MISO communication system intelligent beamforming method based on deep reinforcement learning, by introducing the DDPG model, based on the channel state information (CSI) to jointly optimize the active transmit beamforming matrix of the base station and the reflection coefficient vector of the RIS, so as to maximize the system transmission rate. The purpose of the present invention is achieved through the following technical solutions:

[0007] A RIS-assisted MU-MISO communication system intelligent beamforming method based on deep reinforcement learning comprises the following steps:

[0008] Step 1: Construct a RIS reflection coefficient optimization model with the goal of maximizing multi-user sum rate;

[0009] Specifically, assume that a base station with M antennas provides different communication services for K single-antenna users. The system uses N reflector units (RIS) to assist in enhancing the service transmission performance of the communication system. The direct line-of-sight link between the base station and each user is blocked by an obstacle. and They are represented as complex fading channels from the base station to RIS and from RIS to the kth user respectively; represents the reflection coefficient vector of RIS, where represents the phase shift of the nth reflection unit, j is an imaginary unit; due to the actual hardware limitations, the phase shift of the RIS reflection unit is a discrete phase shift randomly distributed in [0,2π). The bit information b controls one reflection unit, and the reflection unit has a total of B=2 adjustable phase states. b RIS uses an intelligent controller to independently adjust the phase shift of each reflector unit. It is expressed as:

[0010]

[0011] Where Δw = 2π / B; when the signal is transmitted downlink, the base station transmits multiple data streams at the same time, which will interfere with non-target users. The received signal at user k is expressed as:

[0012]

[0013] Where Φ=diag(θ) represents the reflection coefficient matrix of RIS, and the mathematical operator diag(θ) represents extracting the elements of vector θ, and returns a diagonal matrix Φ with the values ​​of θ as diagonal elements; Represents the signal vector transmitted by the base station to K users; the energy of the transmitted signal is normalized, that is, The mathematical operator ||·|| represents the vector 2 norm, is the expectation operator; w k is the kth column vector of the base station active transmit beamforming matrix W, User k receives noise n k The mean is 0 and the variance is complex Gaussian distribution; assuming that all users have the same noise variance, that is, in is the signal that user k expects to receive, It represents the interference signal caused by the service signal of other user n (n≠k) to user k;

[0014] The received signal to interference plus noise ratio (SINR) at the kth user can be expressed as:

[0015]

[0016] Considering the limited transmission power of the base station, under the transmission power threshold constraint, it satisfies:

[0017]

[0018] in, is the transmitted signal power, P t is the transmit power threshold; by jointly designing the active transmit beamforming matrix W of the base station and the reflection unit coefficient matrix Φ of the RIS, a RIS reflection coefficient optimization model is constructed with the goal of maximizing the multi-user sum rate R. The specific optimization problem is expressed as:

[0019]

[0020] stTr{WW H}≤P t , Formula 2

[0021]

[0022] Where R is the multi-user communication rate and;

[0023] Step 2: Train the DDPG model and update the model parameters;

[0024] Specifically, the DDPG model includes training the action network a=q(s|ω a ), target action network a′=q(s′|ω a′ ), training evaluation network Q(s,a|ω c ), target evaluation network Q(s′,a|ω c ) and the experience replay pool; where s represents the state, a represents the action, ω a ,ω a′ ,ω c and ω c′ are the parameters of the corresponding networks respectively;

[0025] During training, first train the evaluation network parameter ω in the DDPG model c Update via:

[0026]

[0027]

[0028] The superscript t indicates that at the tth time, γ c is the learning rate of the training evaluation network, a' is the continuous action output of the target action network, is the corresponding training evaluation network parameter The gradient of , l represents the loss function; and Represent the parameters of the training evaluation network and the target evaluation network respectively; r represents the reward, μ∈(0,1] represents the discount rate; Q represents the state-action value function in deep reinforcement learning;

[0029] Then train the action network parameters ω a The update process is:

[0030]

[0031] where γ a is the learning rate for training the action network; So is the training parameter, s (t) is the input deep neural network state; is the gradient of the target evaluation network for action a, is the gradient of the training action network with respect to its parameters;

[0032] Step 3: The target action network and the target evaluation network are updated using a soft update mechanism, which are expressed as:

[0033]

[0034]

[0035] Among them, τ c and τ a Represent the learning rates for updating the target evaluation network parameters and the target action network parameters respectively;

[0036] Step 4: Apply the DDPG model in the RIS-assisted MU-MISO communication system to adaptively adjust the key elements of reinforcement learning;

[0037] Step 5: When the DDPG model has gone through the set maximum training rounds, it outputs the optimal active transmit beamforming matrix W and the RIS finite discrete phase shift reflection coefficient vector θ.

[0038] Furthermore, the key elements in step 4 include agent, state, action, and reward. The specific operations of adaptive adjustment are as follows:

[0039] Step 401: Agent: RIS controller and base station;

[0040] Step 402: State s at time t (t) The dimension is 2(K+K 2 +N+MK+NM+KN);

[0041] Step 403: Action a at time t (t) Defined as (vec(Re{W (t)},Im{W (t)}),Re{θ (t)},Im{θ (t)}), whose dimension is 2(N+MK); where vec(·) represents the vectorization operation of the matrix by stacking the column vectors of the matrix, Re{·} represents the operation of taking the real part of the complex matrix, and Im{·} represents the operation of taking the imaginary part of the complex matrix;

[0042] Step 404: Reward r at time t (t) Determined according to the user communication rate and the multi-user communication rate of the RIS-assisted MU-MISO communication system;

[0043] Step 405: Initialization, including experience replay pool M, training action network q(s; ω a ), target action network q'(s;π a' ), transmit active transmit beamforming matrix W (0) , training evaluation network Q(s,a;π c ), target evaluation network Q'(s,a;π c' ) and the reflection coefficient vector θ( 0 );

[0044] Step 406: The input of the training action network is the current state, with dimension A in =2(K+K 2 +N+MK+NM+KN); the output is the action taken, which can be expressed as Dimension A out =2(N+MK);

[0045] Step 407: After training the action network output layer, a normalization layer and a quantization layer are introduced in parallel. The working principle of the normalization layer is expressed as follows:

[0046]

[0047] in, Active transmit beamforming matrix to meet transmit power requirements;

[0048] The input of the quantization layer is the continuous phase shift part θ in the output layer cont , and its output is the phase shift θ of the RIS reflection unit coefficient that satisfies the finite discrete phase shift constraint disc ;

[0049] Ensure that the output RIS reflection coefficient phase shift θ disc The range is kept within [0,2π), and the mathematical expression of the scalar quantization layer is expressed as:

[0050]

[0051] where θ disc,n and θ cont,n They are θdisc and θ cont The nth element of , tanh(·) represents the hyperbolic tangent function. i} represents the quantized θ disc,n The specific phase shift value is {a i} represents the steepness of the quantization function. Assuming a i =a,i=1,2,…B-1; {ρ i} represents the quantization region of the quantization function. In the action network training stage, only {ρ i} are learnable parameters.

[0052] Step 408: In the prediction phase, a real quantization layer is constructed according to the learnable parameters of the quantization layer to replace the quantization layer, which can be expressed as:

[0053]

[0054] where ρ0≤ρ11≤…≤ρ B-1 ;

[0055] Step 409: At time t, the system first constructs the current state s (t) , and input it into the action network to produce action a (t) ; After that, the state of the agent is updated to s (t+1) , and calculate the immediate reward according to Formula 1 in the optimization problem:

[0056]

[0057] in for The kth column of for The nth column of Φ disc =diag(θ disc Then, form the experience tuple (s (t) ,a (t) ,r (t) ,s (t+1) ), add it to the experience replay pool. Once the amount of data in the experience replay pool reaches the preset experience replay pool threshold, the network parameters are updated.

[0058] Furthermore, the training action network sequentially comprises an input layer, a fully connected hidden layer, a batch normalization layer, a fully connected hidden layer, a batch normalization layer, and an output layer, and a parallel normalization layer and a quantization layer are introduced after the output layer; wherein the number of neurons in the two fully connected hidden layers is 8A out , 4A out .

[0059] Furthermore, the structure and parameters of the target evaluation network in the DDPG model are directly copied from the training action network, and its parameters remain unchanged during the process of updating the training action network; the network structure and parameters of the target evaluation network are completely copied from the training evaluation network, and the parameters of the target evaluation network remain unchanged during the process of updating the training evaluation network parameters.

[0060] The beneficial effects of the present invention are as follows: the present invention provides a RIS-assisted MU-MISO communication system intelligent beamforming method based on deep reinforcement learning, and proposes a deep deterministic policy gradient (DDPG) method based on channel state information. By interacting with the environment, the beamforming parameters of the communication system are adaptively optimized, thereby achieving optimal transmission performance under actual channel conditions, providing new ideas for the development of future 6G communication systems, and having important application prospects and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the system model of the present invention.

[0062] Figure 2 This is a schematic diagram of the DDPG algorithm framework.

[0063] Figure 3 Schematic diagram of the convergence of the CSI-based DDPG model.

[0064] Figure 4 Schematic diagram of the change of communication speed with the base station transmission power. DETAILED DESCRIPTION

[0065] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0066] In the model of MU-MISO downlink communication system assisted by RIS, as the number of RIS reflection units increases, the computational complexity and required time will become difficult to control. The present invention discloses a deep deterministic policy gradient method for reconfigurable intelligent surface reflection coefficient and beamforming optimization with the goal of maximizing sum rate.

[0067] like Figure 1 As shown in the figure, assume that a base station with M antennas provides different communication services for K single-antenna users. The system uses N reflector units RIS devices to assist in enhancing the service transmission performance of the communication system. The direct line of sight link between the base station and each user is blocked by an obstacle. It is represented as an N×M dimensional complex fading channel between the base station and RIS, It is represented as an N×1 dimensional complex fading channel from RIS to the kth user. represents the reflection coefficient vector of RIS, where Represents the phase shift of the nth reflection unit, j is an imaginary unit. Due to the actual hardware limitations, the phase shift of the RIS reflection unit is a discrete phase shift randomly distributed in [0,2π). The bit information b controls a reflection unit, and the reflection unit can adjust the phase state B = 2 b RIS uses an intelligent controller to independently adjust the phase shift of each reflector unit. It can be expressed as:

[0068]

[0069] Where Δw = 2π / B. Therefore, the phase shift coefficient of the nth RIS reflection unit satisfies When the signal is transmitted downlink, the received signal at user k can be expressed as:

[0070]

[0071] Wherein Φ=diag(θ) represents the reflection coefficient matrix of RIS, and the mathematical operator diag(θ) represents extracting the elements of the vector θ, and returns a diagonal matrix Φ with the element values ​​of θ as diagonal elements. Represents the signal vector transmitted by the base station to K users, and the transmitted signal energy is normalized, that is, The mathematical operator ||·|| represents the vector 2 norm, Is the expectation operator. is the active transmit beamforming matrix of the base station. User k receives noise n k The mean is 0 and the variance is The complex Gaussian distribution of Without loss of generality, we assume that all users have the same noise variance, that is,

[0072] The base station transmits multiple data streams simultaneously, which will cause interference to non-target users:

[0073]

[0074] in is the signal that user k expects to receive, represents the interference signal caused by the service signal of other user n (n≠k) to user k, w k It is the kth column vector of the base station’s active transmit beamforming matrix W.

[0075] The received signal to interference plus noise ratio (SINR) at the kth user can be expressed as:

[0076]

[0077] In addition, considering the limited transmission power of the base station, under the transmission power threshold constraint, it satisfies:

[0078]

[0079] in, is the transmitted signal power, P t is the transmit power threshold (i.e., the maximum transmit power). In order to make full use of the advantages of RIS in the MU-MISO communication system, the active transmit beamforming matrix W of the base station and the reflection unit coefficient matrix Φ of the RIS are jointly designed to construct a RIS reflection coefficient optimization model with the goal of maximizing the multi-user sum rate R. The specific optimization problem is expressed as:

[0080]

[0081] stTr{WW H}≤P t , (b)

[0082]

[0083]

[0084] Where R is the multi-user communication rate and (d) indicates that the phase shift coefficient of any RIS reflection unit satisfies the discrete phase shift set The optimization variables W and θ are coupled to each other, and the above problem is a very challenging non-convex optimization problem. As the number of RIS reflection units increases, the computational complexity and time required will become difficult to control. At present, most studies use alternating iterative optimization algorithms based on convex approximation to find suboptimal solutions. This method first fixes one of W or θ during the iteration process, optimizes the other variable, and then alternates until the problem reaches rate convergence, which requires a lot of time and computing resources.

[0085] DDPG is an extension of the traditional Deep Q-Network (DQN) and is used to solve the deep reinforcement learning algorithm for continuous action space. The algorithm uses a deterministic strategy, that is, directly outputs the value of the action, making the training more stable. In addition, with the help of the action-execution architecture, the action is responsible for learning the strategy, and the Deep Neural Networks (DNN) approximates and learns the local optimal strategy. The same strategy method is used to store past experience through experience replay, and training is performed in random batches to break data correlation, improve sample utilization, and reduce training variance. The specific network structure of DDPG is as follows: Figure 2 shown.

[0086] Agent: An entity that performs tasks and observes the environment, playing a central role in the actual task.

[0087] State: It is defined as a set of observations that reflect the characteristics of the environment and accurately describe the state of the environment at a given point in time. (t) ∩S is used to represent the state observed at time point t, where S represents the state space.

[0088] Action: is a series of choices made by the agent according to a strategy during the learning process. At time t, the agent performs an action a according to the strategy (t) ∈A, A is the sum of the agent action space. The environment state changes from the current s (t) Transition to next state s (t+1) During this process, the environment will also give corresponding reward feedback based on the action of the agent.

[0089] Reward: refers to the feedback that the agent gets from the environment for performing a certain action. It is used to evaluate the utility of the agent performing a specific action in the current state. (t) Represents the agent at time t, state s (t) and perform action a (t) Next, the rewards obtained from the system.

[0090] Strategy: defined as the (t) The next agent chooses action a (t) The probability of (t) ,a (t) ) means to select action a in a given state (t) At the same time,

[0091] State-action value Q function: represents the state s (t) Take action a (t) After that, the potential impact of the environment on the agent is expressed as Q π (s (t) ,a (t) ) indicates. Reward r (t) It measures the immediate reward in a given state, while the value Q function measures the agent’s (t) The potential for future rewards.

[0092] Experience and experience pool: At time point t, an experience tuple contains the current state s (t) , Action taken a (t) 、Instant Rewards (t) And the next moment state s( t+1). The experience pool serves as a warehouse for storing experiences at various time points, providing a large number of samples for training, denoted by M.

[0093] The DDPG model framework includes training action network a=q(s|ω a ), target action network a′=q(s′|ω a′ ), training evaluation network Q(s,a|ω c ), target evaluation network Q(s′,a′|ω c′ ) and the experience replay pool. The training action network has the same structure as the target action network, with the input being the state and the output being the action. However, the training action network inputs the current state s and outputs the action a under the current state; while the target action network inputs the next state s′ and outputs the action a′ under the next state. The training evaluation network has the same structure as the target evaluation network, with the input being the state and action and outputting the Q value of taking the action under this state. The training evaluation network inputs the current state s and the current action a, and outputs the Q value of the current state and the current action; while the target evaluation network inputs the next state s′ and the next action a′, and outputs the Q value of the next state s′ and the next action a′. Training action network q(s|ω a ) is responsible for learning the optimal policy of the agent. The target action network q(s′|ω a′ ) and its parameters are directly copied from the trained action network q(s|ω a ), whose parameters remain unchanged during the update of the training action network to reduce the volatility of the target. Training evaluation network Q(s,a|ω c ) receives the state of the agent and the continuous actions generated by the training action network as input, and outputs the estimated Q value. The target evaluation network Q(s′,a′|ω c′ ) is completely derived from the network structure and parameters of the training evaluation network Q(s,a|ω c ) is copied. In the process of updating the training evaluation network parameters, the target evaluation network parameters remain unchanged to prevent the Q function from not converging; ω a ,ω a′ ,ω c and ω c′ are the parameters of the corresponding networks respectively.

[0094] S1.DDPG specific network parameter update starts with the parameters ω of the training evaluation network c gives:

[0095]

[0096]

[0097] Among them, γ cis the learning rate of the training evaluation network, a' is the continuous action output of the target action network, is the corresponding training evaluation network parameter gradient. and They represent the parameters of the training evaluation network and the target evaluation network respectively. The parameters of the target evaluation network will be updated to the parameters of the training evaluation network at a specific time. The update speed of the target evaluation network parameters is much lower than the update speed of the training evaluation network parameters.

[0098] S2. Then train the action network parameters ω a The update process is:

[0099]

[0100] where γ a is the learning rate for training the action network. So is the training parameter s (t) is the input deep neural network state. is the gradient of the target evaluation network for action a, is the gradient of the training action network with respect to its parameters. The update of the training action network is affected by the target evaluation network with respect to the action gradient. This is done to ensure that the next action selection can be carried out in the direction of maximizing the benefit, thereby optimizing the Q function.

[0101] The Q function is the state-action value function in deep reinforcement learning: it represents the state s (t) Take action a (t) After that, the potential impact of the environment on the agent is expressed as Q π (s (t) ,a (t) ) indicates. Reward r (t) It measures the immediate reward in a given state, while the value Q function measures the agent’s (t) The potential for future rewards.

[0102] In a given state s (t) 、Action a (t) and the immediate reward r at time t (t) In the case of, the Q function is expressed as:

[0103]

[0104] in, Represents the cumulative sum of all immediate rewards after time t, expressed as:

[0105]

[0106] where μ∈(0,1] refers to the discount rate, r (t+τ+1) Represents the timely reward at time t+τ+1. This parameter indicates that earlier behaviors have a smaller impact on the Q function. It aims to evaluate the effect of the expected future cumulative discounted reward brought by selecting a specific action during the learning process.

[0107] Without knowing the exact reward model and state transition model, the optimal Q can be obtained by recursive solution * (s (t) ,a (t) ), the update of the Q function can be expressed as:

[0108]

[0109] Among them, α is the learning rate for updating the Q function.

[0110] In reinforcement learning, the Q function is completely determined by the parameter ω and can be expressed as:

[0111]

[0112] Where ω is the weight and bias parameter in the neural network. Deep reinforcement learning can update ω through a stochastic optimization algorithm to approximate the optimal Q function.

[0113] ω (t+1) =ω (t) -γΔ ω l(ω)

[0114] Among them, Δ ω represents the gradient of the loss function l(ω) with respect to ω, and γ is the learning rate used when updating the parameter ω.

[0115] In reinforcement learning, since the learning process aims to approximate the optimal Q-value function, the true target value is unknown. To address this problem, it is proposed to use two neural networks with the same structure: a training neural network and a target neural network, whose value functions are respectively represented by Q(ω (train) |s (t) ,a (t) ) and Q(ω (target) |s (t) ,a (t) ). The actual target value is estimated to be:

[0116]

[0117] At the same time, the loss function can be expressed as:

[0118] l(ω)=(yQ(ω (train) |s (t) ,a (t))) 2

[0119] S3. In order to maintain the stability of the training process and prevent the training action network from diverging, the target action network and the target evaluation network are updated using a soft update mechanism, which are expressed as:

[0120]

[0121]

[0122] Among them, τ c and τ a Represent the learning rates for updating the target evaluation network parameters and the target action network parameters respectively.

[0123] S4. Applying the DDPG model in the RIS-assisted MU-MISO communication system requires adaptive adjustment of the key elements of reinforcement learning.

[0124] S401. Intelligent agent: RIS controller and base station.

[0125] S402. State: state s at time t (t) The base station's transmission power P at time t t 、The user's received power P r , action a at time t-1( t-1 ) is the active transmit beamforming matrix W( t-1 ) and the RIS reflection coefficient θ( t-1 ), the channel matrix H and Considering that neural networks can only process real number inputs and cannot directly process complex numbers, if complex numbers are involved, their real and imaginary parts need to be split as independent inputs. (t) The dimension is 2(K+K 2 +N+MK+NM+KN).

[0126] S403. Action: Action a at time t (t) The active transmit beamforming matrix W (t) and the RIS reflection coefficient θ (t) Composition, a (t) is defined as (vec(Re{W (t)},Im{W (t)}),Re{θ (t)},Im{θ (t)}), whose dimension is 2(N+MK), vec(·) means stacking the column vectors of the matrix to realize the vectorized operation of the matrix, Re{·} means taking the real part of the complex matrix, and Im{·} means taking the imaginary part of the complex matrix.

[0127] S404. Reward: At time t, reward r (t) It is determined based on the user communication rate and R of the RIS-assisted MU-MISO communication system.

[0128] S405. Initialize the experience replay pool M and train the action network q(s|ω a ), target action network q(s′|ω a′ ), transmit active transmit beamforming matrix W (0) , train the evaluation network Q(s,a|ω c ), target evaluation network Q(s′,a′|ω c′ ) and the reflection coefficient vector θ (0) .

[0129] S406. The input of the training action network is the current state, with dimension A in =2(K+K 2 +N+MK+NM+KN). The output is the action taken, which can be expressed as Dimension A out =2(N+MK). The training action network consists of an input layer, two fully connected hidden layers, two batch normalization layers, an output layer, a normalization layer, and a quantization layer. In order, the number of neurons in the two fully connected hidden layers is 8A. out , 4A out .

[0130] S407. In order to ensure compliance with the transmission power constraint in the optimization problem, the action network introduces a normalization layer, and its working principle can be expressed as:

[0131]

[0132] in, Active transmit beamforming matrix to meet transmit power requirements.

[0133] S408. In order to meet the finite discrete phase shift in the optimization problem The action network uses a quantization layer, and the input of the quantization layer is the continuous phase shift part θ in the output layer. cont , and its output is the phase shift θ of the RIS reflection unit coefficient that satisfies the finite discrete phase shift constraint disc .

[0134] S409. The quantization layer constructs a parameter-learnable differentiable function to approximate the non-differentiable quantization function, that is, the cumulative sum of the shiftable hyperbolic tangent curve is used as the activation function. In order to ensure that the output RIS reflection coefficient phase shift θ discThe range is kept within [0, 2π), and the mathematical expression of the scalar quantization layer is expressed as:

[0135]

[0136] where θ disc，n and θ cont，n They are θ disc and θ cont The nth element of , tanh(·) represents the hyperbolic tangent function. i} represents the quantized θ disc,n The specific phase shift value of disc,n It depends on the number of bits b that control the discrete phase shift of the RIS reflection unit, so we have {a i} represents the steepness of the quantization function. Assuming a i =a, i=1, 2, ... B-1, and is a parameter that can be set manually and cannot be learned. i} represents the quantization region of the quantization function. In the action network training stage, only {ρ i} are learnable parameters.

[0137] S4010. In the prediction stage, a real quantization layer is constructed according to the learnable parameters of the quantization layer to replace the quantization layer, which can be expressed as:

[0138]

[0139] where ρ0≤ρ1≤…≤ρ B-1 .

[0140] S4011. At time point t, the system first constructs the current state s (t) , and input it into the action network to produce action a (t) . After that, the state of the agent is updated to s (t+1) , and calculate the immediate reward according to formula 1 in for The kth column of for The nth column of Φ disc =diag(θ disc Then, form the experience tuple (s (t) ,a (t) ,r (t) ,s (t+1) ), add it to the experience replay pool M. Once the amount of data in the experience replay pool reaches the preset experience replay pool threshold, the network parameters are updated.

[0141] S5. When the network has gone through the set maximum training rounds, it outputs the optimal active transmit beamforming matrix W and the RIS finite discrete phase shift reflection coefficient vector θ.

[0142] In order to verify the convergence of the CSI-based DDPG model, Figure 3 The instantaneous reward and average reward curves of the CSI-based DDPG model over time are shown. In the simulation, the average reward is calculated using the following method:

[0143]

[0144] Where T is the maximum time step, r (i) represents the instantaneous reward at time step i. It can be observed from the figure that the instantaneous reward of the CSI-based DDPG model will fluctuate, and the fluctuation tends to slow down as time i passes. In addition, the CSI-based DDPG model can gradually reduce the average reward r a (t) Converge to a higher sum rate value.

[0145] Figure 4 The graph shows the relationship between the total communication rate and the base station transmission power. In this simulation experiment, the number of base station transmission antennas M = 10, the number of RIS reflection units N = 32, and the number of single-antenna users K = 10. The performance curve marked "BCH-based continuous phase shift algorithm" is the result obtained by the traditional continuous convex optimization approximation method, and the performance curve marked "DQN algorithm" is the result obtained by the traditional deep Q network simulation. It can be seen from the figure that with the increase of base station transmission power, the communication rate of all algorithms also increases accordingly. The figure also shows that under the same RIS reflection unit discrete phase shift bit number condition b, the performance of the CSI-based DDPG model is better than that of DQN. When b = 1 bit, the performance gap between them is about 0.23bit / s / Hz. When b = 2 bits, the performance gap between the CSI-based DDPG model and the benchmark algorithm 1 is about 0.33bit / s / Hz, while the performance gap between the BCH algorithm and DQN is about 0.45bit / s / Hz. In addition, as the number of bits b controlling the discrete phase shift of the RIS reflection unit increases, the discrete phase shift of the RIS has more directions and options, and the communication rate of the system also increases. Compared with the traditional convex optimization approximation iterative approximation algorithm with high-intensity computing resources, the present invention uses reinforcement learning technology to adaptively optimize the beamforming parameters of the communication system through interaction with the environment, which can effectively cope with the rapidly changing channel environment and provide important support for the application of RIS in actual wireless communication systems.

[0146] The above is a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art do not depart from the spirit and scope of the present invention, and should be within the scope of protection of the claims attached to the present invention.

Claims

1. A RIS-assisted MU-MISO communication system intelligent beamforming method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Construct a RIS reflection coefficient optimization model for reconfigurable smart surface with the goal of maximizing multi-user sum rate; Specifically, assume that a base station with M antennas provides different communication services for K single-antenna users. The system uses N reflector units (RIS) to assist in enhancing the service transmission performance of the communication system. The direct line-of-sight link between the base station and each user is blocked by an obstacle. and They are represented as complex fading channels from the base station to RIS and from RIS to the kth user respectively; represents the reflection coefficient vector of RIS, where represents the phase shift of the nth reflection unit, j is an imaginary unit; due to the actual hardware limitations, the phase shift of the RIS reflection unit is a discrete phase shift randomly distributed in [0,2π). The bit information b controls a reflection unit, and the adjustable phase state of the reflection unit is B=2 b RIS uses an intelligent controller to independently adjust the phase shift of each reflector unit. It is expressed as: Where Δw = 2π / B; when the signal is transmitted downlink, the base station transmits multiple data streams at the same time, which will interfere with non-target users. The received signal at user k is expressed as: Where Φ=diag(θ) represents the reflection coefficient matrix of RIS, and the mathematical operator diag(θ) represents extracting the elements of vector θ, and returns a diagonal matrix Φ with the values ​​of θ as diagonal elements; Represents the signal vector transmitted by the base station to K users; the energy of the transmitted signal is normalized, that is, The mathematical operator ||·|| represents the vector 2 norm, is the expectation operator; w k is the kth column vector of the base station active transmit beamforming matrix W, User k receives noise n k The mean is 0 and the variance is complex Gaussian distribution; assuming that all users have the same noise variance, that is, in is the signal that user k expects to receive, It represents the interference signal caused by the service signals of other users n, n≠k to user k; The received signal-to-interference-to-noise ratio at the kth user can be expressed as: Considering the limited transmission power of the base station, under the transmission power threshold constraint, it satisfies: in, is the transmitted signal power, P t is the transmit power threshold; by jointly designing the active transmit beamforming matrix W of the base station and the reflection unit coefficient matrix Φ of the RIS, a RIS reflection coefficient optimization model is constructed with the goal of maximizing the multi-user sum rate R. The specific optimization problem is expressed as: stTr{WW H }≤P t , Formula 2 Where R is the multi-user communication rate and; Step 2: Train the deep deterministic policy gradient DDPG model and update the model parameters; Specifically, the DDPG model includes training the action network a=q(s|ω a ), target action network a′=q(s′|ω a′ ), training evaluation network Q(s,a|ω c ), target evaluation network Q(s′,a′|ω c ′) and the experience replay pool; where s represents the state, a represents the action, ω a ,ω a′ ,ω c and ω c′ are the parameters of the corresponding networks respectively; During training, the training evaluation network parameter ω in the DDPG model c Update via: The superscript t indicates that at the tth time, γ c is the learning rate of the training evaluation network, a' is the continuous action output of the target action network, is the corresponding training evaluation network parameter The gradient of represents the loss function; ω c (target) and Represent the parameters of the training evaluation network and the target evaluation network respectively; r represents the reward, μ∈(0,1] represents the discount rate; Q represents the state-action value function in deep reinforcement learning; Then train the action network parameters ω a The update process is: where γ a is the learning rate for training the action network; So is the training parameter; is the gradient of the target evaluation network for action a, is the gradient of the training action network with respect to its parameters; Step 3: The target action network and the target evaluation network are updated using a soft update mechanism, which are expressed as: Among them, τ c and τ a Represent the learning rates for updating the target evaluation network parameters and the target action network parameters respectively; Step 4: Apply the DDPG model in the RIS-assisted MU-MISO communication system to adaptively adjust the key elements of reinforcement learning; the key elements include agent, state, action, and reward. The specific operations of adaptive adjustment are as follows: Step 401: Agent: RIS controller and base station; Step 402: State s at time t (t) The dimension is 2(K+K 2 +N+MK+NM+KN); Step 403: Action a at time t (t) Defined as (vec(Re{W (t) },Im{W (t) }),Re{θ (t) },Im{θ (t) }), whose dimension is 2(N+MK); where vec(·) represents the vectorization operation of the matrix by stacking the column vectors of the matrix, Re{·} represents the operation of taking the real part of the complex matrix, and Im{·} represents the operation of taking the imaginary part of the complex matrix; Step 404: Reward r at time t (t) Determined according to the user communication rate and the multi-user communication rate of the RIS-assisted MU-MISO communication system; Step 405: Initialization, including experience replay pool M, training action network q(s; ω a ), target action network q'(s;π a′ ), transmit active transmit beamforming matrix W (0) , training evaluation network Q(s,a;π c ), target evaluation network Q′(s,a;π c′ ) and the reflection coefficient vector θ (0) ; Step 406: The input of the training action network is the current state, with dimension A in ,A in =2(K+K 2 +N+MK+NM+KN); the output is the action taken, expressed as Dimension A out ,A out =2(N+MK); Step 407: A normalization layer and a quantization layer are introduced in parallel after the action network output layer. The working principle of the normalization layer is expressed as follows: in, Active transmit beamforming matrix to meet transmit power requirements; The input of the quantization layer is the continuous phase shift part θ in the output layer cont , and its output is the phase shift θ of the RIS reflection unit coefficient that satisfies the finite discrete phase shift constraint disc ; Ensure that the output RIS reflection coefficient phase shift θ disc The range is kept in [0,2π), and the mathematical expression of the scalar quantization layer is expressed as: where θ disc,n and θ cont,n They are θ disc and θ cont The nth element of , tanh(·) represents the hyperbolic tangent function; {c i } represents the quantized θ disc,n The specific phase shift value is {a i } represents the steepness of the quantization function, assuming a i =a,i=1,2,…B-1; {ρ i } represents the quantization region of the quantization function. In the action network training stage, only {ρ i } are learnable parameters; Step 408: In the prediction phase, a real quantization layer is constructed according to the learnable parameters of the quantization layer to replace the quantization layer, which can be expressed as: where ρ0≤ρ1≤…≤ρ B-1 ; Step 409: At time t, the system first constructs the current state s (t) , and input it into the action network to produce action a (t) ; After that, the state of the agent is updated to s (t+1) , and calculate the immediate reward based on the optimization problem formula 1 in for The kth column of for The nth column of Φ disc =diag(θ disc ); Then, form the experience tuple (s (t) ,a (t) ,r (t) ,s (t+1 )), add it to the experience replay pool; once the amount of data in the experience replay pool reaches the preset experience replay pool threshold, the network parameters will be updated; Step 5: After the maximum number of training rounds, DDPG outputs the optimal active transmit beamforming matrix W and RIS finite discrete phase shift reflection coefficient vector θ.

2. According to claim 1, a RIS-assisted MU-MISO communication system intelligent beamforming method based on deep reinforcement learning is characterized in that: The training action network includes an input layer, a fully connected hidden layer, a batch normalization layer, a fully connected hidden layer, a batch normalization layer, and an output layer in sequence. A parallel normalization layer and a quantization layer are introduced after the output layer. The number of neurons in the two fully connected hidden layers is 8A. out , 4A out .

3. According to claim 1, a RIS-assisted MU-MISO communication system intelligent beamforming method based on deep reinforcement learning is characterized in that: The structure and parameters of the target action network in the DDPG model are directly copied from the training action network, and its parameters remain unchanged during the process of updating the training action network; the network structure and parameters of the target evaluation network are completely copied from the training evaluation network, and the parameters of the target evaluation network remain unchanged during the process of updating the training evaluation network parameters.

Images