An active RIS control method based on LSTM-DDPG for EH-NOMA system

Abstract

An active RIS control method based on LSTM-DDPG for an EH-NOMA system comprises the following steps: S1, establishing a communication system model based on energy harvesting and non-orthogonal multiple access for the active RIS system; S2, establishing a Markov process with a RIS controller as an intelligent agent, whose environmental state action reward S3, updating the network parameters of the Critic main network and the Actor main network. The advantages of the invention are: the present application is based on an active RIS communication system model based on energy harvesting and non-orthogonal multiple access, combined with the LSTM-DDPG cascade algorithm to jointly control the power amplification matrix and the phase offset matrix of the active RIS.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning and communication technology, and in particular to an active RIS control method of an EH-NOMA system based on LSTM-DDPG. Background Art

[0002] In order to provide additional cascade channels to both communicating parties in wireless communication, smart reflective surfaces (RIS) can significantly improve the performance of the communication system. RIS is a planar array composed of a large number of passive elements. By reasonably designing the amplitude and phase of these elements through the RIS controller, effective reflection of electromagnetic waves can be achieved, thereby effectively improving the communication quality. However, existing studies on RIS often assume that the direct link quality in the system is poor or the direct link is blocked by obstacles. Because in these scenarios, the noise in the RIS cascade channel is negligible. On the contrary, when the direct link quality in the system is not very poor, the traditional RIS will not be able to achieve effective performance gain due to the multiplicative fading effect of the channel.

[0003] In order to solve this application bottleneck of traditional RIS, the concept of active RIS was proposed. Different from traditional RIS, each component of active RIS is equipped with a power amplifier to amplify the reflected signal. By introducing a power amplifier, amplifying the reflected signal can effectively weaken the multiplicative fading effect.

[0004] Currently, active RIS has the following technical problems:

[0005] 1. Active RIS only considers amplifying the reflected signal without considering the problem of large consumption of energy resources.

[0006] 2. Existing literature related to active RIS does not consider the shortage of spectrum resources in wireless communication systems.

[0007] 3. In the existing literature on the design of the phase offset matrix for RIS, almost all of them assume that the global channel state information (CSI) in the system can be accurately obtained, which is difficult to achieve in practical applications.

[0008] 4. Currently, no existing literature has designed the amplification matrix and phase shift matrix for the EH-NOMA system assisted by active RIS.

[0009] The reasons are as follows:

[0010] 1. In active RIS, in order to amplify the reflected signal, the energy consumed will be much greater than that of passive RIS. This is contrary to the green design concept of RIS, which is low energy consumption and low cost. In addition, in actual application scenarios, RIS is often installed on the surface of high-rise buildings, and it is difficult to provide a stable and reliable energy supply for each RIS component.

[0011] 2. Spectrum resources have always been one of the most precious resources in the field of wireless communication. With the rapid increase in the number of wireless communication devices, the existing spectrum resources can hardly meet the growing user communication needs. Therefore, it is urgent to improve the utilization rate of spectrum resources in various wireless communication systems. The existing literature on active RIS has not considered the shortage of user spectrum resources.

[0012] 3. CSI is an important basis for designing RIS. In order to obtain accurate CSI, RIS needs to have the ability to estimate CSI, that is, to accurately process and analyze pilot signals. However, active RIS has a simple structure and does not have the ability to process and analyze pilot signals.

[0013] In summary, how to design the amplification matrix and phase shift matrix of active RIS without knowing the accurate CSI is an urgent problem to be solved. Summary of the invention

[0014] In order to solve the design problem of the amplification matrix and phase offset matrix of RIS in the EH-NOMA wireless communication system assisted by active RIS, the present invention proposes an active RIS control method based on LSTM-DDPG for the EH-NOMA system. The specific technical scheme is as follows:

[0015] An active RIS control method for an EH-NOMA system based on LSTM-DDPG includes the following steps:

[0016] S1. Establish an active RIS communication system model based on energy harvesting and non-orthogonal multiple access for the active RIS system;

[0017] S2. Establish a Markov process with RIS controller as the intelligent agent, and its environment state action award The environmental state includes the communication state of all users estimated by the LSTM algorithm and the remaining power in the current RIS battery; the actions are the diagonal elements of the power amplification matrix and the phase offset matrix and θ m ,in 0≤θ m <2π,m∈M; in the reward, r is the reward coefficient, which is a positive constant;

[0018] S3. Update the network parameters of the Critic main network and the Actor main network.

[0019] Specifically, the communication system model in step S1 includes a RIS controller as an intelligent agent, a base station, and K user terminals. The intelligent agent receives base station information and sends signals to the user terminal, and the user terminal obtains mixed information forwarded by the base station and the RIS; the intelligent agent is centrally installed on the surface of a high-rise building, and each component in the intelligent agent is provided with an amplifying device to amplify the received signal. The system also includes a power supply for the intelligent agent function.

[0020] Specifically, the environmental status steps in step S2 are as follows:

[0021] SA21. Determine the active RIS communication system model. First, determine the signal received by user K. All users share a spectrum channel resource in NOMA mode. Assume that the communication status of all users in different time slots is dynamically changing. The communication status of all users in the system is dynamically changing. The communication probability of each user is independent and identically distributed, and all obey the "random walk" model. The mixed signal received by user k is:

[0022]

[0023] Among them U k (t) and p k is the communication state of user k at time t and the power of the base station to user k; H s,k is the channel between RIS and user k, h k (t) = h B,k (t)+H S,k (t)PΘH B,S (t) is the equivalent cascade channel from the base station to the RIS and from the RIS to the user k. P and Θ are the RIS power amplification matrix and phase offset matrix to be designed. z represents the noise at the RIS. n k Denote the noise at user k; x k (t) is the signal sent to user k; U j (t) is the binary flag of whether user j is communicating. j (t)=1, otherwise U j (t) = 0; h j (t) is the equivalent cascade channel between user j and the base station, p j (t) is the base station's transmission power to it, x j (t) represents the signal of user j;

[0024] Continuous interference cancellation is performed at each user. After continuous interference cancellation, the interference received by user k is expressed as:

[0025]

[0026] Among them, χ jk (t) is a binary symbol, χ jk (t) = 1 means that the signal strength of user j is stronger than that of user k, otherwise jk (t) = 0, binary symbol d j (t) is used to indicate whether user j is decoded successfully. j (t) = 1 indicates that user j has been successfully decoded, otherwise d j (t) = 0;

[0027] The rate obtained by user k is expressed as:

[0028]

[0029] where g k =U k (t)|h k (t)| 2 p k (t) represents the signal strength of user k. Expressed as the power of the noise at RIS, δ 2 It is represented as the power of the noise at user k;

[0030] SA22, LSTM-based user communication state estimation algorithm, predicts the user's communication status at the next moment.

[0031] Specifically, the energy obtained by the energy harvester EH is stored in the battery. Let the energy in the battery at the beginning of time t be E(t), then at the next moment, the energy in the battery is:

[0032] E(t+1)=min{E(t)+e h (t)-e c (t),E max}

[0033] where e h (t) = E H (t)η and are the energy collected and consumed by RIS at time t, and η is the energy conversion factor.

[0034] Specifically, step SA22 is as follows:

[0035] SA221, build a data set; based on the "random walk" model, generate user communication probability data at several moments. This data set will be used for training and testing of the LSTM network according to the set ratio;

[0036] SA222, divide all data into blocks; divide them into several groups in order, and the data after each group of data is the true prediction result corresponding to this group of data;

[0037] SA223, build an LSTM network model; the LSTM network model contains two layers of LSTM neurons, each LSTM neuron contains three gates, namely the forget gate f(t), the input gate d(t) and the output gate o o (t); the corresponding correlation function is expressed as:

[0038]

[0039]

[0040]

[0041] The output of LSTM is: in

[0042]

[0043]

[0044] Based on the output h(t), the predicted communication probability of the current user is expressed as:

[0045]

[0046] SA224. Train the LSTM network model. The network in the LSTM algorithm is a three-layer network structure, namely the input layer, the LSTM layer and the LSTM output layer. The excitation functions of the LSTM layer and the LSTM output layer are both tanh functions, and the optimizer of the entire network is RMSprop.

[0047] SA225. Apply the LSTM network model; the trained model is evaluated on the untrained user communication status dataset.

[0048] Specifically, the calculation of the action in step S2 is: SB2, output action based on the environmental state; that is, based on the prediction result of the LSTM algorithm, the deep reinforcement learning DRL algorithm-DDPG is used to jointly control the power amplification matrix and phase offset matrix of RIS, and the corresponding RIS power amplification matrix P and phase offset matrix Θ are output. The specific steps are as follows:

[0049] SB21, the agent explores the system environment to obtain experience data; observes the current environment state s(t) at each time slot t, and inputs it into the Actor main network, which outputs the corresponding action a(t) and adjusts the output action. The adjusted action is as follows

[0050]

[0051] in is the energy consumed by RIS when no power amplification is performed. Energy consumed by RIS; Indicates the power of the main Actor network output calculation. and They are respectively represented as the adjusted power action and phase shift action, θ m (t) represents the output of the main Actor network, and L represents the maximum amplification factor. In addition, the action is denoised, that is:

[0052] a(t)=μ(s(t)|θ μ (t))+n o (t)

[0053] Where s(t) represents the state of the system at the current moment, θ μ (t) represents the network parameters of the main Actor network at the current moment, μ(s(t)|θ μ (t)) is expressed as μ (t) strategy, n o (t) is the exploration noise. After executing the adjusted and noised action and system, the agent obtains the reward R(t), and the system environment proceeds to the next state s(t+1). Based on this exploration, an experience data is obtained. Store it in buffer D.

[0054] Specifically, step S3 is as follows:

[0055] S31, sampling from the buffer D to obtain a data set Ω, and selecting a random sampling mechanism;

[0056] S32, update the network parameters of the Critic main network by minimizing the sample set loss function; calculate the loss function to update the network parameters of the Critic main network; calculate the loss function of the Critic main network by back-propagating the TD error as follows:

[0057]

[0058] Where Ω represents the modulus of the training sample, Q(s(i),μ(s(i)|θ μ )|θ Q(i)) indicates that the current Critic main network is in state s(i), and the strategy μ(s(i)|θ μ ), Ω is the corresponding strategy under this parameter, Q'(i) is the target state value function, which is the output of the target Critic network and is calculated by the Bellman equation: Q'(i) = R(i) + γQ'(s(i+1), μ'(s(i+1)|θ μ′ (i))|θ Q′ (i)), where γ is the discount factor, R(i) is the current reward of the system, Q'(s(i+1),μ'(s(i+1)|θ μ' (i))|θ Q' (i)) is the state value function of the target Critic network in state s(i+1) and strategy μ', θ Q' and θ v' are the network parameters of the target Critic network and the target actor network respectively;

[0059] By minimizing the loss function, the network parameters of the Critic main network are updated as follows:

[0060]

[0061] S33. Update the main Actor network parameters by gradient descent; the main Actor network is updated according to the following policy gradient:

[0062]

[0063] S34, soft update target Critic network and target Actor network; soft update of target Critic network and target Actor network is required every C steps

[0064] θ μ' (t) = τ μ θ μ (t)+(1-τ μ )θ μ' (t),

[0065] θ Q' (t) = τ Q θ Q (t)+(1-τ Q )θ Q' (t)

[0066] S35, the main Actor network and the Critic main network are T all The rounds of exploratory training will gradually converge.

[0067] The advantages of the present invention are:

[0068] (1) This application designs an active RIS communication system model based on energy harvesting (EH) and non-orthogonal multiple access (NOMA) for an active RIS system. The active RIS uses EH to realize energy supply, and all users use NOMA to access the same spectrum resources to communicate with the base station (BS), thereby improving spectrum efficiency.

[0069] (2) This application addresses the problem that the active RIS system cannot obtain the system CSI, and designs an active RIS based on an LSTM-based user communication status prediction algorithm and the predicted communication status results.

[0070] (3) For the EH-NOMA active RIS communication system, the deep reinforcement learning (DRL) algorithm is used to jointly design the amplification matrix and phase shift matrix of RIS to further improve the system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is the system model diagram.

[0072] Figure 2 This is the training flow chart of LSTM algorithm.

[0073] Figure 3 This is the data partitioning principle diagram of the LSTM algorithm.

[0074] Figure 4 This is the LSTM network structure diagram.

[0075] Figure 5 This is the training flow chart of the DDPG algorithm.

[0076] Figure 6 The performance comparison chart of different algorithms under different number of users K.

[0077] Figure 7 The performance comparison chart of different algorithms under different RIS elements M.

[0078] Figure 8 The performance comparison chart of different algorithms under different rate thresholds R0. DETAILED DESCRIPTION

[0079] An active RIS control method for an EH-NOMA system based on LSTM-DDPG includes the following steps:

[0080] S1. For the active intelligent reflective surface (RIS) system, an active RIS communication system model based on energy harvesting (EH) and non-orthogonal multiple access (NOMA) is established;

[0081] like Figure 1As shown, the communication system model includes a RIS controller as an intelligent agent, a base station, and K user terminals. The intelligent agent receives base station information and forwards signals to the user terminal, and the user terminal obtains mixed information forwarded by the base station and RIS. The intelligent agent is centrally installed on the surface of a high-rise building to reflect the downlink signal. Each component in the intelligent agent is provided with an amplification device to amplify the received signal. The system also includes a power supply for the intelligent agent function, and the power supply can be clean energy such as solar energy, wind energy, and thermal energy.

[0082] S2. Establish a Markov process with RIS controller as the intelligent agent, and its environment state action award The environmental state includes the communication state of all users estimated by the LSTM algorithm and the remaining power in the current RIS battery; the actions are the diagonal elements of the power amplification matrix and the phase offset matrix and θ m ,in 0≤θ m <2π,m∈M; in the reward, r is the reward coefficient, which is a positive constant and is set to 10 in the present invention. The environment state and action are described below respectively.

[0083] SA2, the optimization steps of environmental status are as follows:

[0084] SA21. Based on the above system, determine the active RIS communication system model. First, determine the signal received by user K. All users use the NOMA mode to share a spectrum channel resource. Assume that the communication status of all users in different time slots is dynamically changing. The communication status of all users in the system is dynamically changing. The communication probability of each user is independent and identically distributed, and all obey the "random walk" model. Therefore, the mixed signal received by user k is:

[0085]

[0086] Among them U k (t) and p k is the communication state of user k at time t and the power of the base station to user k; H s,k is the channel between RIS and user k, h k (t) = h B,k (t)+H S,k (t)PΘH B,S (t) is the equivalent cascade channel from the base station to the RIS and from the RIS to the user k. P and Θ are the RIS power amplification matrix and phase offset matrix to be designed. z represents the noise at the RIS. n kDenote the noise at user k; x k (t) is the signal sent to user k; U j (t) is the binary flag of whether user j is communicating. j (t)=1, otherwise U j (t) = 0; h j (t) is the equivalent cascade channel between user j and the base station, p j (t) is the base station's transmission power to it, x j (t) represents the signal of user j;

[0087] In order to reduce the interference between users, the successive interference cancellation (SCI) technology is performed at each user to separate the signals of different users. Specifically, at each user, the user with the strongest signal will be decoded first, and the signal that fails to be decoded or has not been decoded will cause interference to the user. Therefore, after performing the successive interference cancellation, the interference received by user k is expressed as:

[0088]

[0089] Among them, χ jk (t) is a binary symbol, χ jk (t) = 1 means that the signal strength of user j is stronger than that of user k, otherwise jk (t) = 0, binary symbol d j (t) is used to indicate whether user j is decoded successfully. j (t) = 1 indicates that user j has been successfully decoded, otherwise d j (t) = 0;

[0090] The rate obtained by user k is expressed as:

[0091]

[0092] where g k =U k (t)|h k (t)| 2 p k (t) represents the signal strength of user k. Expressed as the power of the noise at RIS, δ 2 It is represented as the power of the noise at user k.

[0093] To meet the user's quality of service (QoS) requirements, the rate needs to meet R k (t)≥R0;

[0094] The energy obtained by energy harvesting EH is stored in the battery. Let the energy in the battery at the beginning of time t be E(t), then at the next moment, the energy in the battery is:

[0095] E(t+1)=min{E(t)+e h (t)-e c (t),E max}

[0096] in are the energy collected and consumed by RIS at time t, and η is the energy conversion factor.

[0097] SA22, LSTM-based user communication state estimation algorithm, predicts the user's communication status at the next moment;

[0098] like Figure 2 As shown, the method steps are as follows:

[0099] SA221, build a data set; based on the "random walk" model, generate 1000 moments of user communication probability data. This data set will be used for training and testing of the LSTM network in a ratio of 7:3;

[0100] SA222, divide all data into blocks; divide them into groups of 5 in order, and the data after each group of data is the true prediction result corresponding to this group of data. Therefore, 1000 communication probability data will be divided into 995 groups. The schematic diagram is as follows Figure 3 shown.

[0101] SA223, build an LSTM network model; the LSTM network model includes two layers of LSTM neurons, which are used to predict the user communication status, such as Figure 4 As shown, each LSTM neuron contains three gates, namely the forget gate f(t), the input gate d(t) and the output gate o o (t); the corresponding correlation function is expressed as:

[0102]

[0103]

[0104]

[0105] The output of LSTM is: in

[0106]

[0107]

[0108] Based on the output h(t), the predicted communication probability of the current user can be expressed as:

[0109]

[0110] SA224, train the LSTM network model; the network in the LSTM algorithm is a three-layer network structure, namely the input layer, the LSTM layer and the LSTM output layer, wherein the excitation functions of the LSTM layer and the LSTM output layer are both tanh functions, the optimizer of the entire network is RMSprop, and the number of neurons in the three layers are 6, 64, and 1 respectively. The input dimension 6 is composed of the user communication probability of 5 time slots and 1 character representing the input dimension, and the network output is the predicted communication probability of the next time slot of the current user. In the present invention, the number of training epochs is set to 20. The probability of user communication is generated based on "random walk", with a drift rate of 0.0005, a volatility of 0.04, and a starting communication probability of 0.6.

[0111] SA225, apply LSTM network model; in order to verify the real effect of LSTM algorithm in actual scenarios, the final trained model needs to be evaluated on the untrained user communication status data set to finally confirm the effect of LSTM network model. In the test, the overlap rate of predicted communication probability and real communication probability data is observed. The prediction results include communication status and other link status information at RIS.

[0112] SB2, output action based on environmental state; that is, based on the prediction results of LSTM algorithm, use deep reinforcement learning DRL algorithm-DDPG to jointly control the power amplification matrix and phase offset matrix of RIS, output the corresponding RIS power amplification matrix P and phase offset matrix Θ, and further improve the system performance; Figure 3 As shown, the specific steps are as follows:

[0113] SB21, the agent explores the system environment to obtain experience data; observes the current environment state s(t) at each time slot t, and inputs it into the Actor main network, which outputs the corresponding action a(t) and adjusts the applicability of the output action based on the system application. The adjusted action is

[0114]

[0115] in is the energy consumed by RIS when no power amplification is performed. Energy consumed by RIS; Indicates the power of the main Actor network output calculation. and They are respectively represented as the adjusted power action and phase shift action, θ m (t) represents the output of the main Actor network, L represents the maximum amplification factor, and in order to fully explore the environment, it is necessary to add noise to the action, that is:

[0116] a(t)=μ(s(t)|θ μ (t))+n o (t)

[0117] Where s(t) represents the state of the system at the current moment, θμ(t) represents the network parameters of the main Actor network at the current moment, and μ(s(t)|θ μ (t)) is expressed as μ (t) strategy, n o (t) is the exploration noise. After executing the adjusted and noised action and system, the agent obtains the reward R(t), and the system environment proceeds to the next state s(t+1). Based on this exploration, an experience data is obtained. Store it in buffer D for training the Actor main network.

[0118] S3. Update the network parameters of the Critic main network and the Actor main network. The specific steps are as follows:

[0119] S31, sampling from the buffer D to obtain a data set Ω, the present invention considers the complexity of the algorithm and the fairness of sample sampling, and selects a random sampling mechanism;

[0120] S32, update the network parameters of the Critic network by minimizing the sample set loss function; calculate the loss function to update the network parameters of the Critic main network; calculate the parameters of the Critic main network when the parameters are θ by back-propagating the TD error Q The loss function is as follows:

[0121]

[0122] Where Ω represents the modulus of the training sample, Q(s(i),μ(s(i)|θ μ )|θ Q (i)) indicates that the current Critic main network is in state s(i), and the strategy μ(s(i)|θ μ ), Ω is the corresponding strategy under this parameter, Q'(i) is the target state value function, which is the output of the target Critic network, and can be calculated by the Bellman equation: Q'(i) = R(i) + γQ'(s(i+1), μ'(s(i+1)|θ μ' (i))|θ Q'(i)), where γ is the discount factor, R(i) is the current reward of the system, Q'(s(i+1),μ'(s(i+1)|θ μ' (i))|θ Q' (i)) is the state value function of the target Critic network in state s(i+1) and strategy μ', θ Q' and θ μ' are the network parameters of the target Critic network and the target actor network respectively.

[0123] By minimizing the loss function, the network parameters of the Critic main network can be updated as follows:

[0124]

[0125] S33. Update the parameters of the main Actor network by gradient descent. The optimization goal of the main Actor network is to obtain the maximum state-action Q value. Therefore, considering that the Q value function is differentiable and the action space is continuous, the main Actor network can be updated according to the following policy gradient:

[0126]

[0127] S34, soft update target Critic network and target Actor network; in order to maintain the stability of algorithm training, the target Critic network and target Actor network need to be soft updated every C steps

[0128] θ μ' (t) = τ μ θ μ (t)+(1-τ μ )θ μ′ (t),

[0129] θ Q′ (t) = τ Q θ Q (t)+(1-τ Q )θ Q′ (t)

[0130] Where 0≤τ μ ≤1 and 0≤τ Q ≤1 are the update factors of the target Critic network and the target Actor network respectively.

[0131] S35, the main Actor network and the Critic main network are T all The rounds of exploratory training will gradually converge.

[0132] In order to verify the performance of the proposed algorithm, several benchmark algorithms are introduced to compare the performance with the proposed LSTM-DDPG algorithm. Among them, the Actor Critic (AC) algorithm is another classic reinforcement learning algorithm. Its network structure has two fewer target networks than the DDPG network, including the Actor network and the Critic network. The performance measurement indicator of different algorithms is the average communication success rate of all users in the system, which is defined as When the rate of user k meets the QoS rate threshold, that is: R k When (t)≥R0,ζ k =1, otherwise ζ k = 0. In order to verify the robustness of the algorithm in this application, Figure 6 , Figure 7 and Figure 8 The performance comparison chart of different algorithms under different numbers of users K, different numbers of RIS elements M, and different rates R0. From the above simulation results, it can be seen that under different system parameters, the proposed LSTM-DDPG algorithm can obtain the best average success rate.

[0133] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An active RIS control method for an EH-NOMA system based on LSTM-DDPG, characterized in that: The following steps are involved: S1. Establish an active RIS communication system model based on energy harvesting and non-orthogonal multiple access for the active RIS system; S2. Establish a Markov process with RIS controller as the intelligent agent, and its environment state action award The environmental state includes the communication state of all users estimated by the LSTM algorithm and the remaining power in the current RIS battery; the actions are the diagonal elements of the power amplification matrix and the phase offset matrix and θ m ,in 0≤θ m <2π,m∈M; r in the reward is the reward coefficient, which is a positive constant; E(t) is the current remaining energy at RIS; and θ m are the mth diagonal elements of the power amplification matrix P and the phase shift matrix θ respectively; is the number of users whose rate is greater than the threshold R0 at time t, that is, the number of users who successfully communicate, and a(t) refers to the action output by the neural network; S3. Update the network parameters of the Critic main network and the Actor main network.

2. According to claim 1, an active RIS control method of an EH-NOMA system based on LSTM-DDPG is characterized in that: The communication system model in step S1 includes a RIS controller as an intelligent agent, a base station, and K user terminals. The intelligent agent receives base station information and sends signals to the user terminal, and the user terminal obtains mixed information forwarded by the base station and the RIS; the intelligent agent is centrally installed on the surface of a high-rise building, and each component in the intelligent agent is provided with an amplifying device to amplify the received signal. The system also includes a power supply for powering the intelligent agent.

3. According to claim 2, an active RIS control method of an EH-NOMA system based on LSTM-DDPG is characterized in that: The environmental status steps in step S2 are as follows: SA21. Determine the active RIS communication system model. First, determine the signal received by user K. All users share a spectrum channel resource in NOMA mode. Assume that the communication status of all users in different time slots is dynamically changing. The communication status of all users in the system is dynamically changing. The communication probability of each user is independent and identically distributed, and all obey the "randomwalk" model. The mixed signal received by user k is: Among them U k (t) and p k is the communication state of user k at time t and the power of the base station to user k; H s，k is the channel between RIS and user k, h k (t) = h B,k (t)+H S，k (t)PΘH B，S (t) is the equivalent cascade channel from the base station to the RIS and from the RIS to the user k. P and Θ are the RIS power amplification matrix and phase offset matrix to be designed. z represents the noise at the RIS. n k Denote the noise at user k; x k (t) is the signal sent to user k; U j (t) is the binary flag of whether user j is communicating. j (t)=1, otherwise U j (t) = 0; h j (t) is the equivalent cascade channel between user j and the base station, p j (t) is the base station's transmission power to it, x j (t) is the signal of user j; h B,k (t) Direct channel between base station and user k, H B，S (t) is the channel between the base station and RIS, H S，k (t) is the channel between RIS and user k; Continuous interference cancellation is performed at each user. After continuous interference cancellation, the interference received by user k is expressed as: Among them, χ jk (t) is a binary symbol, χ jk (t) = 1 means that the signal strength of user j is stronger than that of user k, otherwise jk (t) = 0, binary symbol d j (t) is used to indicate whether user j is decoded successfully. j (t) = 1 indicates that user j has been successfully decoded, otherwise d j (t) = 0; The rate obtained by user k is expressed as: where g k =U k (t)|h k (t)| 2 p k (t) represents the signal strength of user k. Expressed as the power of the noise at RIS, δ 2 It is represented as the power of the noise at user k; SA22, LSTM-based user communication state estimation algorithm, predicts the user's communication status at the next moment.

4. According to claim 3, an active RIS control method of an EH-NOMA system based on LSTM-DDPG is characterized in that: The energy obtained by energy harvesting EH is stored in the battery. Let the energy in the battery at the beginning of time t be E(t), then at the next moment, the energy in the battery is: E(t+1)=min{E(t)+e h (t)-e c (t),E max } where e h (t) = E H (t)η and are the energy collected and consumed by RIS at time t, η is the energy conversion factor; H B，S (t) is the channel between the base station and RIS, E H (t) is the wireless radio frequency energy collected by RIS at time t, η is the energy conversion factor, e h (t) is the usable energy after conversion, E max is the RIS battery capacity, i.e. its maximum storage capacity.

Images