Energy-saving optimization method in STAR-RIS auxiliary communication network based on deep reinforcement learning

Abstract

The invention provides an energy-saving optimization method in an STAR-RIS auxiliary communication network based on deep reinforcement learning. In order to solve the problem that STAR-RIS element switch power consumption is often ignored, a novel STAR-RIS element switch state sensing power consumption model is introduced. The model comprises a system model, a signal model and a power consumption model. The invention provides a deep reinforcement learning algorithm TD3 and an enhanced version TCD3 algorithm thereof, the TCD3 fuses the principles of Clipped Double Q-learning and Maxmin-Q learning, value prediction of three evaluation networks is adopted, and the underestimation problem of the TD3 algorithm is effectively solved under the condition of not introducing extra hyper-parameters. The two algorithms can significantly improve the energy efficiency of the system, and especially TCD3 is superior to TD3 in the aspects of learning speed, average reward, variance reduction and the like.

Description

Technical Field

[0001] This invention belongs to the field of communications, specifically relating to an energy-saving optimization method in STAR-RIS assisted communication networks based on deep reinforcement learning. Background Technology

[0002] With the rapid deployment of fifth-generation (5G) intelligent wireless communication systems, user communication demands have surged, leading to an explosive growth in the number of terminal products. Simultaneously, the world is transitioning to sustainable energy development, making the establishment of green and energy-efficient wireless networks crucial. Therefore, energy efficiency has become a core performance evaluation indicator. Furthermore, due to the high cost and energy consumption of auxiliary communication equipment, one direction in communication energy efficiency research has shifted to finding auxiliary communication equipment that is energy-efficient, intelligent, reliable, and cost-effective.

[0003] Among numerous emerging technologies, Reconfigurable Smart Surfaces (RIS) have shown great potential for reducing the energy consumption of communication systems. RIS achieves this by digitally controlling electromagnetic waves, creating a parameter-tunable electromagnetic environment, including amplitude, phase, polarization, frequency, and delay; this has pioneered a new paradigm for intelligent programmable wireless environments. However, the coverage of traditional RIS is limited, prompting research to shift towards the more advanced STAR-RIS. Building upon the advantages of traditional RIS, STAR-RIS achieves 360-degree omnidirectional coverage by dividing the incident signal into forward reflection and backward transmission. This design allows transmitters and receivers to be positioned on either side of the STAR-RIS, effectively overcoming geographical limitations and achieving comprehensive spatial coverage.

[0004] However, current research often overlooks the energy consumption differences between the on and off states of components on STAR-RIS, an oversight that fails to accurately reflect the characteristics of actual physical components. To address this issue, our paper proposes a novel STAR-RIS component switch-state-aware power consumption model to establish a more accurate energy efficiency (EE) optimization model. Considering the fractional nature of the objective function and the complexity of the switching mechanism, traditional convex optimization methods are insufficient. Meanwhile, deep reinforcement learning has become an important decision-making tool for handling complex problems in wireless networks. Inspired by this, we construct the optimization problem as a Markov decision process, explicitly defining the state space, action space, and reward function, and employ deep reinforcement learning algorithms (TD3 and TCD3) to find the optimal solution. Experimental results show that our proposed algorithms effectively address the non-convexity property of the objective function, with the TCD3-based algorithm achieving a better reward. Summary of the Invention

[0005] This invention provides an energy-saving optimization method for STAR-RIS assisted communication networks based on deep reinforcement learning. The communication system includes a system model, a signal model, and a power consumption model. The optimization problem is expressed in mathematical form, and the problem is optimized using the deep reinforcement learning TD3 algorithm and an enhanced version of the TCD3 algorithm.

[0006] STAR-RIS operates under the Energy Splitting (ES) model, supporting simultaneous signal transmission and reflection. According to the ES protocol, the total energy of the signals transmitted and reflected by STAR-RIS should equal the energy it receives. The amplitude coefficient of each element on STAR-RIS follows the rules... Where n = 1, 2, ..., N. and Let n represent the transmission amplitude coefficient and reflection amplitude coefficient of the nth element, respectively. The energy consumption of STAR-RIS is determined by the characteristics and resolution of its reflective elements, which are crucial for modulating the phase of the incident signal. Considering the energy consumption associated with adjusting the phase shift values ​​of these elements, effectively managing their on / off states yields higher energy efficiency. The ON or OFF state of a STAR-RIS element is denoted by n, and is defined as follows:

[0007]

[0008] The vector representation of the ON / OFF state of a STAR-RIS element is: σ = [σ1, ..., σ N When a STAR-RIS element is in the OFF state, it will not reflect signals from the BS. Therefore, considering the ON / OFF states of the STAR-RIS elements, the coefficient matrix on the STAR-RIS elements is defined as follows:

[0009]

[0010] (2) Signal Model

[0011] This invention uses s k The active beamforming vector represents the information sequence transmitted by the base station to user k.

[0012] The signal transmitted at time t is represented as:

[0013]

[0014] Gaussian noise denoted as n0, user The received signal is represented as:

[0015]

[0016] Similarly, users The received signal is represented as:

[0017]

[0018] Signal-to-noise ratio (SINR) for users R and T and Represented as:

[0019]

[0020] Where σ 2 This represents noise power. Given bandwidth B, the achievable data rate for each user is:

[0021] R k =Blog2(1+γ) k )

[0022] The summation rate for all users is calculated as follows:

[0023]

[0024] (3) Power consumption model

[0025] The overall power utilization of the STAR-RIS auxiliary system studied includes the transmission power from the BS, the circuit power usage of the BS and all connected users, and the energy consumption of STAR-RIS. The total power of the system is calculated as follows:

[0026]

[0027] (4) Problem Statement

[0028] Within the communication system framework of this invention, the objective is to simultaneously optimize the coefficient matrix of STAR-RIS, the beamforming vector of the base station, and the on / off vectors of components on STAR-RIS to improve system energy efficiency while adhering to constraints of minimum data rate and total power limits. The optimization problem of STAR-RIS is mathematically expressed as follows:

[0029]

[0030] w H w≤P max

[0031]

[0032] This invention also proposes two deep reinforcement learning (DRL) algorithms: TD3 and its enhanced version TCD3. Both algorithms use the same state, action, and reward settings.

[0033] (1) Action vector: For the t-th training step, the action vector is constructed using the beamforming vector. coefficient matrix Θ τ,(t) and STAR-RIS switch vector a (t) .because Since Θ is a complex vector, and neural networks require real-valued inputs, it is decomposed into real and imaginary components. The same decomposition applies to the diagonal element Θ. τ,(t) The action vector for the t-th training step is represented as:

[0034]

[0035] 2) State Vector: For each time slot, the state vector St includes Channel State Information (CSI) from the base station (BS) to STAR-RIS, from the BS to T users, and from the base station to T and R users, capturing the current system state and optimization variables. It also includes the action of the previous time step (t-1), the transmission power at the current time step t, and the STAR-RIS switching vector, defined as:

[0036]

[0037] 3) Reward Function: The reward function needs to consider turning off unnecessary STAR-RIS elements to encourage the shutdown of unnecessary components and improve energy efficiency. Here, λ represents the additional reward obtained for turning off each STAR-RIS element, and N is the total number of STAR-RIS elements. Therefore, the reward in the t-th training step is defined as follows:

[0038]

[0039] (2) To address the underestimation bias caused by the TD3 algorithm, we introduce a reinforcement method called Triplet Critic Update. This method combines the principles of Clipped Double Q-learning and Maxmin-Q-learning, employing value predictions from three evaluation networks without introducing additional hyperparameters. The Triplet Critic Update mechanism first selects the larger value from two evaluators, then selects the smaller value from that value and a third independent evaluator. When this strategy is integrated into the TD3 framework, it forms the TCD3 algorithm. The update rules of TCD3 are defined as follows:

[0040]

[0041] Beneficial effects

[0042] This invention analyzes a STAR-RIS-assisted MISO communication network, deploying a STAR-RIS to assist communication between a base station and single-antenna users. Within this framework, user pairs are divided into two groups, located in the transmission and reflection areas of the STAR-RIS, respectively. To achieve more precise energy efficiency optimization in the STAR-RIS-assisted communication system, this invention introduces an innovative switch-aware power consumption model. For this model system, an optimization problem is established with the goal of maximizing energy efficiency.

[0043] This invention introduces two optimization algorithms based on deep reinforcement learning. In the TD3 algorithm, the minimum Q-value estimated by two evaluators is used, which systematically underestimates the Q-value. To address this issue, this invention proposes a triple evaluation update method, which combines the principles of Clipped Double Q-learning and Maxmin-Q learning, employing value predictions from three evaluation networks without introducing additional hyperparameters. This is the TCD3 algorithm, and this innovation has not yet been applied to the field of DRL-based wireless communication optimization. Experimental results show that our proposed algorithm effectively solves the non-convexity property of the objective function, with the TCD3-based algorithm achieving a better reward. This highlights its superior ability to optimize complex wireless communication scenarios. Specific advantages will be explained in detail below with reference to the accompanying figures. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the system model of the STAR-RIS assisted wireless network in this invention.

[0045] Figure 2 This is a schematic diagram of the TCD3 network structure in this invention. Detailed Implementation

[0046] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings of the embodiments of the present invention. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0047] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0048] This invention provides an energy-saving optimization method for STAR-RIS assisted communication networks based on deep reinforcement learning, which includes: a system model, a signal model, a power consumption model, and a problem formulation. Two deep reinforcement learning (DRL) algorithms are also proposed: TD3 and its enhanced version TCD3.

[0049] (1) System Model

[0050] This invention designs a multiple-input single-output (MISO) communication system, which includes a multi-antenna base station (BS) and a reconfigurable smart surface (STAR-RIS) for synchronized transmission and reflection, serving multiple single-antenna users. The base station and STAR-RIS are fixed in specific locations, consistent with the common practice of installing STAR-RIS on the exterior walls of tall buildings. Users are classified into transmission zone (T) users and reflection zone (R) users, denoted by I and U respectively, and these users are allowed to move within a certain range. It is assumed that the direct link between users in the reflection zone and the base station is blocked by buildings, while other links remain unobstructed.

[0051] STAR-RIS employs an Energy Split (ES) model, which supports simultaneous transmission and reflection of elements. According to the ES protocol, the total energy of the signals transmitted and reflected by STAR-RIS should be equal to the energy received. We have established amplitude coefficient rules for each element. n = 1, 2, ..., N. Where, and These represent the transmission and reflection amplitude coefficients of the nth element, respectively. The energy consumption of STAR-RIS is determined by the characteristics and resolution of its elements, which are key factors in modulating the phase of the incident signal. Given the energy overhead of adjusting the phase shift of these elements, strategically managing their on / off states yields higher energy efficiency. The on / off states of STAR-RIS elements are determined by the vector σ. n It indicates and has the following characteristics:

[0052]

[0053] The vector representing the on / off state of a STAR-RIS element is labeled as: σ = [σ1, ..., σ N When an element on STAR-RIS is in the off state, it does not reflect or refract signals from the base station. Therefore, considering the on / off states of STAR-RIS elements, we can define the coefficient matrix on STAR-RIS as follows:

[0054]

[0055] Where τ∈{T,R}, the phase is represented by... Let represent the element located in the interval [0, 2π). Let n represent the nth element on STAR-RIS, where n ∈ {1, 2, ..., N}.

[0056] (2) Signal Model

[0057] This invention uses s k The active beamforming vector represents the information sequence transmitted by the base station to user k.

[0058] The signal transmitted at time t is represented as:

[0059]

[0060] Gaussian noise denoted as n0, user The received signal is represented as:

[0061]

[0062] Similarly, for user D's received signal for:

[0063]

[0064] user and The signal-to-noise ratio (SINR) is expressed as:

[0065]

[0066] Where σ 2 This represents the noise power. For any user, given a bandwidth B, the achievable data rate is:

[0067] R k =Blog2(1+γ) k )

[0068] The summation rate for all users is calculated as follows:

[0069]

[0070] (3) Power consumption model

[0071] The total power consumption of the communication system studied in this invention includes the transmission power of the base station, the circuit power consumption of the base station and all receiving users, and the energy consumption of STAR-RIS. The total power of the system is calculated as follows:

[0072]

[0073] Where μ = ν -1 In the figure, ν represents the power amplifier efficiency of the base station. P BP represents the circuit power consumption of the base station. k P represents the circuit power consumption for any user. R This indicates the power consumption of each reflective element in STAR-RIS.

[0074] (4) Problem Formulation

[0075] In this invention, our goal is to simultaneously optimize the coefficient matrix of STAR-RIS, the beamforming vector of the base station, and the on / off vectors of components on STAR-RIS to improve system energy efficiency while adhering to constraints of minimum data rate and total power limits. The optimization problem of STAR-RIS is mathematically expressed as follows:

[0076]

[0077] w H w≤P max

[0078]

[0079] Where R min P represents the minimum data rate required by any user. max This represents the maximum allowable transmission power of the base station. Constraints 3 and 4 ensure that the amplitude and phase shift coefficients of the elements on STAR-RIS are within a reasonable range. Vector σ represents the on / off state of the STAR-RIS elements.

[0080] This invention also proposes two deep reinforcement learning (DRL) algorithms: TD3 and its enhanced version TCD3.

[0081] (1)TD3

[0082] The TD3 algorithm improves upon the traditional deep learning algorithm, DDPG (Deep Deterministic Policy Gradient), by introducing three key techniques to address its overestimation bias problem. First, TD3 employs clipped double-Q learning, where the temporal difference (TD) objective effectively reduces the overestimation bias in the traditional DDPG method by calculating the minimum of two target value networks. Second, TD3 introduces random noise into the target policy to smooth action selection. This noise comes from a truncated normal distribution, ensuring that excessive variables are not introduced into the action output. Third, TD3 adjusts the update frequency of the policy and target networks, choosing a conservative update strategy. The update frequency of the policy and target networks is lower than that of the value network, updating once every k iterations, where k is a hyperparameter to ensure that the policy update is based on a reliable value estimate.

[0083] The TD3 algorithm consists of six deep neural networks: two value networks, two corresponding target value networks, one policy network, and one target policy network. The architecture of the value and policy networks in TD3 includes an input layer, two hidden layers, and an output layer. The value network takes the state and action (s, a) as input and outputs a scalar Q-value, where Q represents the expected value of taking action a in state s. The policy network takes state s as input, outputs action a, and scales it appropriately according to the constraints of the action space. The output layer uses the tanh activation function to constrain the action value within a normalized range, ensuring that the action is feasible in the environment.

[0084] TD3, as an off-policy algorithm, allows experiences collected under any behavioral policy to be stored in an experience replay buffer. These experiences are then used to train the policy network and the value network, where the experiences are expressed as transformation tuples (s t ,a t ,r t ,s t+1 The parameters of the target network are stored in the form of a policy network and randomly sampled for training the neural network. The policy network is updated by policy gradients, while the value network is trained by minimizing the loss function of the TD error. The target network gradually aligns its parameters with those of the main network through soft updates.

[0085] 1) Action Vector: In the t-th training step, the action vector is derived from the beamforming vector. STAR-RIS coefficient matrix Θ τ,(t) and the switching vector σ (t) Composition, due to Since Θ is a complex vector, and neural networks require real inputs, we decompose it into real and imaginary parts. The coefficient matrix Θ τ,(t) The diagonal elements are also decomposed in the same way. The action vector in the t-th training step is represented as:

[0086] Where Re{·} and Im{·} represent the real part and the imaginary part, respectively. It is the nth diagonal element of the matrix.

[0087] 2) State vector: The state vector S for each time slot t t It includes Channel State Information (CSI) from the base station to STAR-RIS, CSI from the base station to user T, and CSI from STAR-RIS to users T and R, and includes the action of the previous step (t-1), the transmission power of the current step, and the STAR-RIS switching vector, defined as follows:

[0088]

[0089] 3) Reward Function: The reward function needs to consider turning off unnecessary STAR-RIS elements to encourage the shutdown of unnecessarily used components and improve energy efficiency. Therefore, the reward in the t-th training step is defined as follows:

[0090]

[0091] Here, λ represents the additional reward gained for disabling a STAR-RIS element, and N is the total number of STAR-RIS elements. The condition penalizes the reward when any user's data rate falls below a set minimum, thus maintaining the specified constraints. Conversely, when all users' data rates meet the standard, the condition keeps the reward at its original value. This mechanism allows the TD3 model to improve its energy efficiency (EE) value by training and fine-tuning its parameters, avoiding negative rewards and unnecessary overhead.

[0092] (2) Enhanced version TCD3

[0093] To address the underestimation bias caused by the TD3 algorithm, this invention introduces a reinforcement method called Triplet Critic Update. This method combines the principles of Clipped Double Q-learning and Maxmin-Q-learning, employing value predictions from three evaluation networks without introducing additional hyperparameters. The Triplet Critic Update mechanism first selects the larger value from two evaluators, then selects the smaller value from that value and a third independent evaluator. When this strategy is integrated into the TD3 framework, it forms the TCD3 algorithm (Triplet Critic Delayed DeepDeterministic Policy Gradient). The update rules for TCD3 are defined as follows:

[0094]

[0095] The TCD3 algorithm effectively reduces underestimation and provides more accurate value estimates without introducing additional hyperparameters. Simulation experiments show that TCD3 provides more accurate value estimates than TD3, ensuring more reliable results in reinforcement learning applications. Specific details of the TCD3 method are provided in Algorithm 1 and Algorithm 2. The TCD3 algorithm uses the same settings for state, action, and reward as the TD3 algorithm described above.

[0096]

[0097]

[0098] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. This invention provides an energy-saving optimization method for STAR-RIS assisted communication networks based on deep reinforcement learning, comprising: System model, signal model, power consumption model, and problem formulation.

2. The energy-saving optimization method in a STAR-RIS assisted communication network based on deep reinforcement learning as described in claim 1, characterized in that: In the system model, STAR-RIS operates under an energy splitting (ES) model, supporting simultaneous transmission and reflection. According to the ES protocol, the total energy of the signals transmitted and reflected by STAR-RIS should equal the energy it receives. The amplitude coefficient rule for each element is established as follows: Where n = 1, 2, ..., N. and Let n represent the transmission amplitude coefficient and reflection amplitude coefficient of the nth element, respectively. The energy consumption of STAR-RIS is determined by the characteristics and resolution of its reflective elements, which are crucial for modulating the phase of the incident signal. Considering the energy consumption associated with adjusting the phase shift values ​​of these elements, strategically managing their on / off states leads to higher energy efficiency. The ON or OFF state of a STAR-RIS element is denoted by n, and its characteristics are as follows: The ON / OFF state vector of a STAR-RIS element is represented as: σ = [σ1, ..., σ N When a STAR-RIS element is in the OFF state, it does not reflect signals from the BS. Therefore, considering the ON / OFF states of the STAR-RIS elements, the present invention can define the coefficient matrix on the STAR-RIS elements as follows:

3. The energy-saving optimization method in a STAR-RIS assisted communication network based on deep reinforcement learning as described in claim 1, characterized in that: Use s k The active beamforming vector represents the information sequence transmitted by the base station to user k. The signal transmitted at time t is represented as: The Gaussian noise is denoted by n0. The signal received by user R in the reflection zone is: Similarly, the received signal for user T in the refraction region is: Signal-to-noise ratio (SINR) for users R and T and Represented as: Where σ 2 This represents noise power. Given bandwidth B, the achievable data rate for each user is: R k =Blog2(1+c k ) The total rate for all users is:

4. The energy-saving optimization method in a STAR-RIS assisted communication network based on deep reinforcement learning as described in claim 1, characterized in that: The total power of the STAR-RIS auxiliary system studied in the power consumption model includes the transmission power from the BS, the circuit power of the BS and all connected users, and the energy consumption of STAR-RIS. The total power of the system is calculated as follows:

5. The energy-saving optimization method in a STAR-RIS assisted communication network based on deep reinforcement learning as described in claim 1, characterized in that: The objective of this invention is to simultaneously optimize the coefficient matrix of STAR-RIS, the beamforming vector of the base station, and the on / off vectors of components on STAR-RIS to improve system energy efficiency while adhering to constraints of minimum data rate and total power limits. The optimization problem of STAR-RIS is mathematically expressed as follows: st In H in≤P max 6. This invention also proposes two deep reinforcement learning (DRL) algorithms: TD3 and its enhanced version TCD3. Both algorithms use the same state, action, and reward settings. (1) Action vector: In the t-th training step, the action vector is derived from the beamforming vector. STAR-RIS coefficient matrix Θ τ,(t) and the switching vector σ (t) Composition, due to Since Θ is a complex vector, and neural networks require real inputs, we decompose it into real and imaginary parts. The coefficient matrix Θ τ,(t) The diagonal elements are also decomposed in the same way. The action vector in the t-th training step is represented as: 2) State vector: The state vector S for each time slot t t It includes Channel State Information (CSI) from the base station to STAR-RIS, CSI from the base station to user T, and CSI from STAR-RIS to users T and R, and includes the action of the previous step (t-1), the transmission power of the current step, and the STAR-RIS switching vector, defined as follows: 3) Reward Function: The reward function involves disabling each unnecessary STAR-RIS element to improve energy efficiency while satisfying system constraints. Here, λ represents the additional reward obtained for disabling each STAR-RIS element, and N is the total number of STAR-RIS elements. The reward for training at step t is defined as:

7. The energy-saving optimization method in a STAR-RIS assisted communication network based on deep reinforcement learning as described in claim 1, characterized in that: The enhanced TCD3 algorithm is called Triple Evaluator Update. This method integrates the principles of Clipped Double Q-learning and Maxmin-Q learning, effectively constraining the value predictions of the three evaluators without introducing additional hyperparameters. Figure 2 shows the network structure of the TCD3 algorithm. The triple evaluation update mechanism first selects the larger value from two evaluators, then selects the smaller value from that value and a third independent evaluator. When this strategy is integrated into the TD3 framework, it forms the TCD3 algorithm. The update rule of TCD3 is defined as follows:

Images