Multi-intelligent metasurface enabled MISO system precoding and phase shift joint design method

Abstract

The invention relates to a precoding and phase shift joint design method for a multi-intelligent metasurface enabled MISO (Multiple Input Single Output) system. The method comprises the following steps: S1, constructing a multi-intelligent metasurface enabled MISO system model; and S2, designing base station BS precoding and intelligent metasurface RIS phase shift based on TD3. According to the method, system spectrum efficiency maximization under base station BS precoding and multi-intelligent metasurface phase constraint under an actual phase model is achieved, the TD3-based base station BS precoding matrix and intelligent metasurface RIS phase joint design scheme can significantly improve the spectrum efficiency of the system, and therefore the spectrum efficiency can be maximized, and the system performance is improved. And a solid foundation is laid for deploying the multiple intelligent metasurfaces in the MISO system.

Description

[0001] Field

[0002] The present invention relates to a mobile communication method, in particular to a multi-intelligent metasurface-enabled MISO system precoding and phase shift joint design method. Background Art

[0003] With the popularization of millimeter wave frequency bands in 5G cellular systems, the demand for spectrum and energy efficiency of existing communication technologies continues to increase. However, due to its short wavelength characteristics, the millimeter wave frequency band often encounters problems such as high power consumption and signal attenuation during data transmission, which poses a severe challenge to achieving effective transmission. In this context, smart metasurface RIS, as an emerging technology, can improve the performance of the system by reshaping the signal transmitted by the base station BS. With its significant advantages such as low cost, energy saving and environmental protection, and high gain, smart metasurface RIS technology is regarded as a highly promising technology in the future 6G communication field. Smart metasurface RIS is composed of a large number of low-cost passive components. By adjusting the phase of these components, the reflection angle of the incident electromagnetic signal is improved, thereby improving the system performance.

[0004] Drawing on the advantages of intelligent metasurface RIS, many scholars have introduced intelligent metasurface RIS into the multiple-input single-output system (MISO) to improve the performance of the MISO system. However, the algorithm based on alternating optimization is too complex in the real intelligent metasurface RIS-assisted communication system, making it difficult to ensure real-time decision-making. In addition, most of the current considerations are single-intelligent metasurface RIS-assisted MISO systems, and the transmission efficiency of multi-intelligent metasurfaces in multi-user MISO systems has not been studied. How to efficiently apply multi-intelligent metasurfaces to multi-user MISO downlink transmission systems has not yet been well solved. With the widespread application of millimeter wave frequency bands and the increase in network complexity, how to deploy multi-intelligent metasurfaces to MISO systems to maximize the system's spectrum efficiency is a problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a joint design method of precoding and phase shift of a MISO system enabled by a multi-intelligent metasurface, so as to apply the multi-intelligent metasurface to the MISO system and maximize the system spectrum efficiency.

[0006] The object of the present invention is achieved in this way;

[0007] A method for joint design of precoding and phase shift of a MISO system enabled by multi-intelligent metasurfaces comprises the following steps:

[0008] S1. Construct a MISO system model enabled by multi-intelligent metasurfaces. The specific method is as follows:

[0009] The system described in S1-1 is that L intelligent supersurfaces RIS are deployed within the coverage of a base station BS to provide auxiliary services for K user equipments UE; the base station BS is equipped with M antennas, and each user equipment UE is equipped with a single antenna; each user equipment UE is provided with auxiliary services by the intelligent supersurface RIS closest to it; the number of user equipments UE served by the lth intelligent supersurface RIS is g l ; The set G of the number of user equipment UEs served by all intelligent metasurface RIS is: G = {g1, g2, ..., g L}.

[0010] The total number K of all user equipments UE served by the base station BS is:

[0011] The number of reflective elements N on each intelligent metasurface RIS is: N = N x *N y ;

[0012] Among them, N x N is the number of horizontal rows of the smart metasurface RIS array; y is the number of vertical columns of the smart metasurface RIS array.

[0013] S1-2 In the system, the signal y received at the kth user equipment UE k It is expressed as:

[0014]

[0015] Among them, n k represents the additive white Gaussian noise at the kth user equipment UE; x k represents the signal sent by the base station BS to the kth user equipment UE; w k represents the precoding vector of the signal of the kth user equipment UE; k represents the equivalent channel coefficient between the base station BS and the kth user equipment UE.

[0016] Equivalent channel coefficient h k For: h k =h r,k +h t,k ;

[0017] Among them, h r,k represents the composite channel from the base station BS to the kth user equipment UE reflected by the intelligent metasurface RIS; h t,k represents the channel between the base station BS and the kth user equipment UE.

[0018] Channel h t,k Following the mmWave SV channel model:

[0019]

[0020] Among them, PL t,k represents the path loss factor; N t,k represents the number of non-line-of-sight clusters from the base station BS to the kth user equipment UE; L t,k represents the number of paths in each cluster; N t,l represents the number of non-line-of-sight clusters from the base station BS to the lth smart metasurface RIS; L t,l represents the number of paths in each cluster; represents the gain of the qth path in the ith cluster represents the departure angle at the base station BS, d is the distance between the transmitter and the receiver, and λ is the carrier wavelength.

[0021] Composite channel h r,k It is expressed as:

[0022]

[0023] Among them, H t,l is the channel from the base station BS to the lth smart metasurface RIS; h l,k is the channel from the lth smart metasurface RIS to the kth user equipment UE.

[0024] The SV channel matrix from the S1-3 base station BS to the lth intelligent metasurface RIS is expressed as:

[0025]

[0026] The SV channel matrix from the lth intelligent metasurface RIS to the kth user equipment UE is expressed as:

[0027]

[0028] Among them, PL t,l represents the link loss factor from the base station BS to the lth smart metasurface RIS; PL l,k G represents the link loss factor from the lth intelligent metasurface RIS to the kth user equipment UE; RIS,l is the gain compensation at the lth smart metasurface RIS; (N t,l and L t,l ) are the number of clusters; (N l,k and L l,k ) are the number of paths in the corresponding cluster; α i,q and Each is the gain of the qth path in the ith cluster of a link; let the 0th cluster in formula (4) represent the line-of-sight path of each link; represents the departure angle at the base station BS; is the azimuth of the path to the smart metasurface RIS; is the pitch angle of the path to the smart metasurface RIS; is the azimuth angle of the RIS path leaving the smart metasurface; is the pitch angle of the RIS path leaving the smart metasurface.

[0029] S1-4 Phase shift matrix Θ of each smart metasurface RIS l for:

[0030]

[0031] Among them, θ l,n is the phase shift of the nth reflective element of the lth smart metasurface RIS; β l,n is the amplitude coefficient of the nth reflective element of the lth smart metasurface RIS.

[0032] S1-5 Amplitude coefficient β l,n With phase shift θ l,n The functional relationship between them is approximately:

[0033]

[0034] Among them, θ l,n ∈[θ min ,θ max ],θ min is the lower bound of the phase shift under hardware constraints, θ max is the upper limit of the phase shift under hardware limitations; α is the parameter that controls the steepness of the function curve; β min is the minimum amplitude; φ0 is the difference between the minimum amplitude phase and -π; when α=0, β l,i =1, which is the ideal case of reflection amplitude.

[0035] S1-6 The achievable rate C of the kth user equipment UE k It is expressed as:

[0036]

[0037] Among them, w k represents the precoding vector of the signal of the kth user equipment UE; k represents the equivalent channel coefficient between the base station BS and the kth user equipment UE; σ is the standard deviation parameter of the noise.

[0038] S1-7 Multi-intelligent metasurface assisted multi-user downlink system spectral efficiency maximization is expressed as:

[0039]

[0040] sttr(WW H )≤P max #(7b)

[0041] θ l,n ∈[θ min ,θ max ]#(7c)

[0042] Where W = [ω1,ω2,…,ω K ] is the precoding matrix, W H is the conjugate matrix of the precoding matrix; P max is the maximum transmission power of the base station BS; θ l,n is the phase shift of the nth reflective element of the lth smart metasurface RIS; θ min is the lower bound of the phase shift under hardware constraints, θ max It is the upper limit of the phase shift under hardware limitations.

[0043] S2. Design of BS precoding and intelligent metasurface RIS phase shift based on TD3, the specific method is as follows:

[0044] S2-1 defines the environmental state as: and

[0045] Among them, a t-1 represents the action at time t-1; r t-1,k is the achievable rate of the kth user equipment UE at time t-1.

[0046] The optimization parameters (W, Θ) in the action space l ) is defined as:

[0047]

[0048] in, represents the transpose of the precoding vector of the signal of the Kth user equipment UE; θ L,N is the phase shift of the Nth reflective element of the Lth smart metasurface RIS.

[0049] The reward function is set as:

[0050] The S2-2 TD3 algorithm consists of six networks: an Actor current network, an Actor target network, two Critic current networks, and two Critic target networks; the parameter of the Actor current network is φ; the parameter of the Critic current network 1 is θ1; the parameter of the Critic current network 2 is θ2; the Critic current network 1 is s t As input; Critic current network 2 with at is the input and outputs the corresponding Q value, which is defined as:

[0051]

[0052] Among them, r(s t ,a t ) represents the immediate reward r of the model t ;γ is the discount factor; s t is the environmental state; a t For action; t+1 The updated environment status.

[0053] In performing action a t After that, the environment state is updated to s t+1 . Use the experience replay pool to store experience (s t ,a t ,r t ,s t+1 ), and randomly extract several batches of samples from them to calculate the loss value and update the current network of Critic. S2-3 The loss function L(θ of the current network of Critic i )for:

[0054]

[0055] Among them, y t is the target Q value; r t represents the immediate reward of the model; s t is the environmental state; a t For action.

[0056] S2-4 selects a smaller Q value as the estimated Q value to reduce the overestimation of the Q value, and adds noise ∈ to the target strategy to avoid overfitting; when updating the Actor current network, its gradient is:

[0057]

[0058] Among them, s t is the environmental state; a t For action.

[0059] S2-5 updates the parameters of the Actor's current network and the Critic's target network through soft update:

[0060] θ′=τθ+(1-τ)θ′#(11)

[0061] φ′=τφ+(1-τ)φ′#(12)

[0062] Among them, τ is the soft update coefficient, θ is the parameter of the Critic target network, θ′ is the parameter of the Critic target network after update, φ is the parameter of the Actor current network, and φ′ is the parameter of the Actor current network after update.

[0063] The present invention realizes the maximization of system spectrum efficiency under the constraints of base station BS precoding and multi-intelligent metasurface phase under the actual phase model by constructing the MISO system network architecture enabled by multi-intelligent metasurfaces; the joint design scheme of base station BS precoding matrix and intelligent metasurface RIS phase based on TD3 can significantly improve the spectrum efficiency of the system, thereby maximizing the spectrum efficiency and laying the foundation for the deployment of multi-intelligent metasurfaces in MISO systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a schematic diagram of the multi-intelligent metasurface-assisted MISO system.

[0065] Figure 2 This is the TD3 algorithm framework diagram.

[0066] Figure 3 It is a pseudo code diagram of the TD3 spectrum efficiency maximization transmission algorithm. DETAILED DESCRIPTION

[0067] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0068] The joint design method of precoding and phase shift of MISO system enabled by multi-intelligent metasurfaces of the present invention comprises the following steps: S1, constructing a MISO system model enabled by multi-intelligent metasurfaces; S2, designing base station BS precoding and intelligent metasurface RIS phase shift based on TD3. The specific description is as follows:

[0069] S1. Construct a MISO system model enabled by multi-intelligent metasurfaces. The specific method is as follows:

[0070] S1-1 Figure 1 As shown in FIG. 1 , the MISO system deploys L intelligent metasurfaces RIS within the coverage of a base station BS to provide auxiliary services for K user equipment UEs, wherein the base station BS is equipped with M antennas and each UE is equipped with a single antenna. Each UE is provided with auxiliary services by the RIS closest to it, and the number of UEs served by the lth RIS is g. l , the number of UEs served by all RIS is defined as a set G, expressed as G = {g1,g2,…,g L}. The number of all UEs served by the BS is Each RIS contains N=N x *N y reflective elements to further enhance the signal strength.x ,N y are the number of horizontal rows and vertical columns of the RIS array respectively.

[0071] S1-2 In the constructed MISO system, the signal y received at the kth UE k It can be expressed as:

[0072]

[0073] in, represents the additive white Gaussian noise at the kth UE; σ 2 represents variance; x k represents the signal sent by the BS to the kth UE; w k represents the precoding vector of the signal of the kth UE; h k =h r,k +h t,k represents the equivalent channel coefficient between the BS and the kth UE, where h r,k represents the composite channel reflected from the BS to the kth UE via RIS, h t,k represents the channel between the BS and the kth UE, which follows the millimeter wave Saleh-Valenzuela (SV) channel model and is expressed as follows:

[0074]

[0075] Among them, PL t,k represents the path loss factor; N t,k represents the number of non-line-of-sight clusters from the base station BS to the kth UE; L t,k represents the number of paths in each cluster; N t,l represents the number of non-line-of-sight clusters from the base station BS to the lth RIS; L t,l represents the number of paths in each cluster; represents the gain of the qth path in the ith cluster; a M (θ) is the response vector of the uniform linear array; represents the departure angle at the BS; d represents the distance between the transmitter and the receiver; λ represents the carrier wavelength.

[0076] The composite channel from the BS to the kth UE reflected by RIS is expressed as:

[0077]

[0078] in, and They are the channel from BS to the lth RIS and the channel from the lth RIS to the kth UE respectively.

[0079] The SV channel matrix from the S1-3 base station BS to the lth RIS is expressed as:

[0080]

[0081] The SV channel matrix from the lth RIS to the kth UE is expressed as:

[0082]

[0083] Among them, PL t,l and PL l,k They represent the link loss factors from BS to the lth RIS and from the lth RIS to the kth UE respectively; G RIS,l is the gain compensation at the lth RIS; (N t,l ,L t,l ) and (N l,k ,L l,k ) are the number of clusters and the number of paths in each cluster respectively; and is the gain of the qth path in the ith cluster of the two links. Assume that the 0th cluster in equation (4) represents the line-of-sight path of each link. M,N (φ,θ) represents the response vector of the uniform planar array; represents the departure angle at BS; are the azimuth and elevation angles of the path to RIS, are the azimuth and elevation of the path leaving RIS, respectively.

[0084] S1-4 Each RIS phase shift matrix Θ l for:

[0085]

[0086] Where diag(·) represents the diagonal operation of the matrix; θ l,n ∈[-π,π],β l,n ∈(0,1] are the phase shift and amplitude coefficient of the nth reflector element of the lth RIS respectively.

[0087] S1-5 takes into account the actual relationship between the amplitude coefficient and the phase shift, namely the amplitude coefficient β l,n is the phase shift θ l,n function, which can be approximated as

[0088]

[0089] Among them, θ l,n ∈[θ min ,θ max ], α is the parameter that controls the steepness of the function curve, β minis the minimum amplitude; φ0 is the difference between the minimum amplitude phase and -π; θ min and θ max are the lower and upper bounds of the phase shift under hardware constraints. It is worth noting that when α = 0, β l,i =1, which is the ideal case of reflection amplitude.

[0090] S1-6 The achievable rate C of the kth UE k It is expressed as:

[0091]

[0092] Among them, w k represents the precoding vector of the signal of the kth user equipment UE; k represents the equivalent channel coefficient between the base station BS and the kth user equipment UE; σ is the standard deviation parameter of the noise.

[0093] S1-7 Multi-RIS assisted multi-user downlink system spectrum efficiency maximization can be expressed as:

[0094]

[0095] sttr(WW H )≤P max #(7b)

[0096] θ l,n ∈[θ min ,θ max ]#(7c)

[0097] Where W = [ω1,ω2,…,ω K ] is the precoding matrix, W H is the conjugate matrix of the precoding matrix; tr(·) is the trace of the matrix; P max is the maximum transmission power of BS; θ l,n is the phase shift of the nth reflective element of the lth smart metasurface RIS; θ min is the lower bound of the phase shift under hardware constraints, θ max It is the upper limit of the phase shift under hardware limitations.

[0098] S2. Design of BS precoding and RIS phase shift based on TD3, the specific method is as follows:

[0099] S2-1 Define environmental state s t for:

[0100] and

[0101] Among them, r t-1,kis the achievable rate of the kth UE at time t-1; a t-1 Represents the action at time t-1.

[0102] Action: Since neural networks only consider real values, the input complex-valued data is divided into real and imaginary parts. Therefore, the optimization parameters (W, Θ) in the action space are l ) can be defined as:

[0103]

[0104] in, represents the transpose of the precoding vector of the signal of the Kth UE; θ L,N is the phase shift of the Nth reflective element of the Lth RIS; the size of the action space is 2(MK+LN).

[0105] Therefore, w can be calculated k ←u k w k , k=1,…,K. In order to satisfy constraints (7b) and (7c), the output normalized variable and Considering the actual RIS reflection model, Need to be further combined with amplitude:

[0106]

[0107] Among them, β l,n According to formula (5), in the ideal RIS reflection model,

[0108] Reward: The purpose of this invention is to maximize the spectral efficiency of the multi-intelligent metasurface assisted MISO system, so the reward function is set as:

[0109]

[0110] S2-2 is a BS precoding and RIS phase shift design based on TD3. Its framework is as follows Figure 2 The TD3 algorithm includes six networks: an Actor current network, an Actor target network, two Critic current networks, and two Critic target networks. In the DRL model of the present invention, the parameter of the Actor current network is φ, and the parameters of the Critic current network 1 and the Critic current network 2 are θ1 and θ2 respectively. The Critic current network 1 and the Critic current network 2 are s t and a t is the input and outputs the corresponding Q value, which is defined as:

[0111]

[0112] Among them, r(s t ,a t ) represents the immediate reward r of the model t ;γ is the discount factor; s t is the environmental state; a t For action; t+1 The updated environment status.

[0113] In performing action a t After that, the environment state is updated to s t+1 . Use the experience replay pool to store experience (s t ,a t ,r t ,s t+1 ), and randomly extract a certain batch of samples from it to calculate the loss value and update the current network of Critic. S2-3 The loss function L(θ of the current network of Critic i )for:

[0114]

[0115] Among them, y t is the target Q value; r t represents the immediate reward of the model; s t is the environmental state; a t For action.

[0116] S2-4 selects a smaller Q value as the estimated Q value to reduce the overestimation of the Q value, and adds noise ∈ to the target strategy to avoid overfitting. When updating the Actor's current network, its gradient is:

[0117]

[0118] Among them, s t is the environmental state; a t For action.

[0119] S2-5 updates the parameters of the Actor's current network and the Critic's target network parameters through soft update:

[0120] θ′=τθ+(1-τ)θ′#(11)

[0121] φ′=τφ+(1-τ)φ′#(12)

[0122] Among them, τ is the soft update coefficient, 0≤τ≤1; θ is the parameter of the Critic target network, θ′ is the parameter of the Critic target network after update, φ is the parameter of the Actor current network, and φ′ is the parameter of the Actor current network after update.

[0123] The present invention proposes a pseudo code of a transmission algorithm based on TD3 spectrum efficiency maximization in a multi-RIS assisted MISO system. Figure 3 shown.

[0124] The operation process of the TD3 algorithm is divided into three stages: data collection, model training and parameter adjustment.

[0125] In the data collection phase, the algorithm randomly selects a series of states and inputs them into the Actor network to generate a t 、Action a t , Reward n and subsequent states n+i These quadruple are then stored in the experience replay pool. Once the experience replay pool reaches the upper limit of capacity, the algorithm will enter the training phase; otherwise, continue to sample data.

[0126] After entering the training phase, the algorithm extracts batches of states s from the experience replay pool t 、Action a t , Reward t and subsequent states t+i The sample set is used as the input data for network training. Then the action a can be obtained t+i , and based on this, the target Q value is calculated, and then the timing difference error is obtained.

[0127] Finally, in the parameter adjustment phase, the algorithm follows the delayed update strategy and first updates the parameters of the two Critic online networks. After the Critic1 and Critic2 networks have completed the G-step update, the parameters of the Actor online network are updated. In addition, the soft update technology is used to fine-tune the parameters of the two Critic target networks and the Actor target network to ensure the smoothness and stability of the learning process.

[0128] After the TD3 model training is stable, an efficient BS precoding and RIS phase shift strategy is obtained. After running this strategy, the tiling efficiency of the multi-intelligent metasurface assisted MISO system can be significantly improved.

Claims

1. A joint design method for precoding and phase shifting of a MISO system enabled by multi-intelligent metasurfaces, characterized in that: The following steps are involved: S1. Construct a MISO system model enabled by multi-intelligent metasurfaces. The specific method is as follows: The system described in S1-1 is that L intelligent supersurfaces RIS are deployed within the coverage of a base station BS to provide auxiliary services for K user equipments UE; the base station BS is equipped with M antennas, and each user equipment UE is equipped with a single antenna; each user equipment UE is provided with auxiliary services by the intelligent supersurface RIS closest to it; the number of user equipments UE served by the first intelligent supersurface RIS is g l ; The set G of the number of user equipment UEs served by all intelligent metasurface RIS is: G = {g1, g2, ..., g L }; The total number K of all user equipments UE served by the base station BS is: The number of reflective elements N on each intelligent metasurface RIS is: N = N x *N y ; Among them, N x is the number of horizontal rows of the smart metasurface RIS array; N y is the number of vertical columns of the smart metasurface RIS array; S1-2 In the system, the signal y received at the kth user equipment UE k It is expressed as: Among them, n k represents the additive white Gaussian noise at the kth user equipment UE; x k represents the signal sent by the base station BS to the kth user equipment UE; w k represents the precoding vector of the signal of the kth user equipment UE; k represents the equivalent channel coefficient between the base station BS and the kth user equipment UE; Equivalent channel coefficient h k For: h k =h r，k +h t,k ; Among them, h r，k represents the composite channel from the base station BS to the kth user equipment UE reflected by the intelligent metasurface RIS; h t，k represents the channel between the base station BS and the kth user equipment UE; Channel h t，k Following the mmWave SV channel model: Among them, PL t，k represents the path loss factor; N t，k represents the number of non-line-of-sight clusters from the base station BS to the kth user equipment UE; L t，k represents the number of paths in each cluster; N t，l represents the number of non-line-of-sight clusters from the base station BS to the ,th smart metasurface RIS; L t，l represents the number of paths in each cluster; represents the gain of the qth path in the ith cluster represents the departure angle at the base station BS, d is the distance between the transmitter and the receiver, and λ is the carrier wavelength; Composite channel h r,k It is expressed as: Among them, H t，l is the channel from the base station BS to the lth smart metasurface RIS; h l，k is the channel from the lth smart metasurface RIS to the kth user equipment UE; The SV channel matrix from the S1-3 base station BS to the RIS is expressed as: The SV channel matrix from the kth intelligent metasurface RIS to the kth user equipment UE is expressed as: Among them, PL t，l represents the link loss factor from the base station BS to the ,th smart metasurface RIS; PL l，k G represents the link loss factor from the lth intelligent metasurface RIS to the kth user equipment UE; RIS，l is the gain compensation at the lth smart metasurface RIS; N t，l and L t，l N is the number of clusters; l，k and L l，k are the number of paths in the corresponding cluster; α i，q and Each is the gain of the qth path in the ith cluster of a link; let the 0th cluster in formula (4) represent the line-of-sight path of each link; represents the departure angle at the base station BS; is the azimuth of the path to the smart metasurface RIS; is the pitch angle of the path to the smart metasurface RIS; is the azimuth angle of the RIS path leaving the smart metasurface; is the pitch angle of the RIS path leaving the smart metasurface; S1-4 Phase shift matrix Θ of the lth smart metasurface RIS l for: Among them, θ l，n is the phase shift of the nth reflective element of the RIS smart metasurface; β l，n is the amplitude coefficient of the nth reflective element of the lth smart metasurface RIS; S1-5 Amplitude coefficient β l，n With phase shift θ l，n The functional relationship between them is approximately: Among them, θ l，n ∈[θ min ,θ max ],θ min is the lower bound of the phase shift under hardware constraints, θ max is the upper limit of the phase shift under hardware limitations; α is the parameter that controls the steepness of the function curve; β min is the minimum amplitude; φ0 is the difference between the minimum amplitude phase and -π; S1-6 The achievable rate C of the kth user equipment UE k It is expressed as: Among them, w k represents the precoding vector of the signal of the kth user equipment UE; k represents the equivalent channel coefficient between the base station BS and the kth user equipment UE; σ is the standard deviation parameter of the noise; S1-7 Multi-intelligent metasurface assisted multi-user downlink system spectral efficiency maximization is expressed as: s.t.tr(WW H )≤P max #(7b) i l，n ∈[θ min ,the max ]#(7c) Among them, W=[ω1,ω2,…,ω K ] is the precoding matrix, W H is the conjugate matrix of the precoding matrix; P max is the maximum transmission power of the base station BS; θ l，n is the phase shift of the nth reflective element of the RIS smart metasurface; θ min is the lower bound of the phase shift under hardware constraints, θ max is the upper limit of the phase shift under hardware limitations; S2. Design of BS precoding and intelligent metasurface RIS phase shift based on TD3, the specific method is as follows: S2-1 Define environmental state s t for: and Among them, a t-1 represents the action at time t-1; r t-1，k is the achievable rate of the kth user equipment UE at time t-1; the optimization parameters (W, Θ) in the action space l ) is defined as: in, represents the transpose of the precoding vector of the signal of the Kth user equipment UE; θ L，N is the phase shift of the Nth reflective element of the Lth smart metasurface RIS; The reward function is set as: The S2-2 TD3 algorithm consists of six networks: an Actor current network, an Actor target network, two Critic current networks, and two Critic target networks; the parameter of the Actor current network is φ; the parameter of the Critic current network 1 is θ1; the parameter of the Critic current network 2 is θ2; the Critic current network 1 is s t As input; Critic current network 2 with a t is the input and outputs the corresponding Q value, which is defined as: Among them, r(s t , a t ) represents the immediate reward r of the model t ;γ is the discount factor; s t is the environmental state; a t For action; t+1 is the updated environmental status; In performing action a t After that, the environment state is updated to s t+1 ; Use the experience replay pool to store experience (s t , a t , r t ,s t+1 ), and randomly extract several batches of samples from them to calculate the loss value and update the current Critic network; S2-3 Critic’s current network loss function L(θ i )for: Among them, y t is the target Q value; r t Represents the immediate reward of the model; s t is the environmental state; a t for action; S2-4 selects a smaller Q value as the estimated Q value to reduce the overestimation of the Q value, and adds noise ∈ to the target strategy to avoid overfitting; when updating the Actor current network, its gradient is: Among them, s t is the environmental state; a t for action; S2-5 updates the parameters of the Actor's current network and the Critic's target network through soft update: θ′=τθ+(1-τ)θ′#(11) φ′=τφ+(1-τ)φ′#(12) Among them, τ is the soft update coefficient, θ is the parameter of the Critic target network, θ′ is the parameter of the Critic target network after update, φ is the parameter of the Actor current network, and φ′ is the parameter of the Actor current network after update.

Images