A ris-assisted multi-user MISO system beamforming optimization method

Abstract

This invention relates to the field of telecommunications technology, specifically to a beamforming optimization method for a RIS-assisted multi-user MISO system. First, a RIS-assisted multi-user downlink MISO communication system model is constructed, closely matching the actual transmission characteristics of block fading channels. The system transmission process is accurately characterized through cascaded equivalent channels. Then, a high-dimensional non-convex joint optimization problem is constructed with the goal of maximizing the total system rate. This problem is transformed into a Markov decision process in a continuous action space. An improved TD3 algorithm is used, employing independent Q-value estimation from multiple independent commentator networks and truncated mean aggregation to process the target Q-value. This effectively suppresses Q-value overestimation while avoiding the overly conservative problem caused by the minimum strategy. This significantly improves the stability of algorithm training and the robustness of strategy learning, while simultaneously achieving coordinated optimization of active beamforming by the base station and passive phase shifting by the RIS.

Description

Technical Field

[0001] This invention relates to the field of telecommunications technology, specifically to a beamforming optimization method for a RIS-assisted multi-user MISO system. Background Technology

[0002] Reconfigurable Intelligent Surface (RIS) is a novel programmable electromagnetic metasurface technology that can reconstruct the wireless propagation environment without the need for additional RF links and active amplification by jointly controlling the phase (and amplitude) of a large number of low-cost reflective elements. Therefore, RIS is widely considered one of the key candidate technologies supporting high-speed, low-latency, and green communication in 6G. To this end, we introduce RIS into multi-user MISO (Multiple-Input Single-Output) systems to effectively improve channel conditions, reduce interference, and provide higher signal quality. By jointly optimizing the active beamforming of the base station and the passive phase-shifting matrix of the RIS, we maximize system throughput or improve overall service quality.

[0003] While Deep Reinforcement Learning (DRL) offers a promising paradigm for joint beamforming of Reinforcement-Relationships (RIS), existing actor-critic-based methods still have significant shortcomings. First, as the number of RIS elements and users increases, the dimension of continuous actions increases significantly, making value estimation by a single critic or a small number of critics prone to large biases, leading to instability during training or even convergence failure. Second, to mitigate Q-value overestimation, some algorithms employ a minimum strategy for calculating the target Q. While this suppresses overestimation to some extent, it often introduces excessive conservatism, thus limiting final performance and slowing down the learning speed. Summary of the Invention

[0004] The purpose of this invention is to provide a beamforming optimization method for a RIS-assisted multi-user MISO system.

[0005] The technical solution of this invention is as follows: A beamforming optimization method for a RIS-assisted multi-user MISO system includes the following operations: S1. Construct a RIS-assisted multi-user downlink MISO communication system model, including a base station with M transmit antennas, a RIS with L reconfigurable reflection units, and K single-antenna users, where M≥K; Under block fading channel conditions, establish the base station RIS channel matrix and the RIS user channel matrix, and combine them with the RIS reflection matrix to obtain the cascaded equivalent channel reflected by the RIS. S2. Based on the cascaded equivalent channel, establish a user received signal model, derive the signal-to-interference-plus-noise ratio of each user, and construct a high-dimensional non-convex joint optimization problem of the system with the goal of maximizing the total system rate. The decision variables of the joint optimization problem include at least the base station beamforming matrix and the RIS reflection matrix. S3. The joint optimization problem of the system is transformed into a Markov decision process in a continuous action space. The state includes channel information and control information, the action includes the base station beamforming matrix and RIS reflection coefficient, and the reward is related to the instantaneous total rate of the system. An improved TD3 algorithm is used to train the Markov decision process and learn the policy, and the base station beamforming matrix and RIS reflection matrix are output as the joint beamforming result. The improved TD3 algorithm includes the use of multiple independent commentator networks, each of which learns independently and provides a target Q-value estimate. The truncated mean of all target Q-values ​​is then aggregated to update the target Q-value.

[0006] In S3, the truncated mean aggregation operation is as follows: sort the target Q-value sets of all commenters' network outputs, remove the smallest first-order target Q-values, and take the mean of the remaining target Q-values ​​to obtain the truncated mean target Q-values.

[0007] In the Q-value estimation stage of S3, a random subset sampling mechanism is used. Specifically, during each update of the critic network and the policy network, M critics are randomly selected from the N target critic networks to form a critic subset, where M is a positive integer greater than 2 and M≤N, and this subset is used for target Q-value estimation.

[0008] In S2, the user's signal-to-interference-plus-noise ratio (SIR) is calculated using the following formula: , , , For the first k Signal-to-interference-to-noise ratio for each user For users k The expected signal power For the normalized equivalent noise term under multi-user parallel transmission, RIS user channel matrix The element in the k-th column, For the RIS reflection matrix, For the base station RIS channel matrix, Base station beamforming matrix The Middle k Each element also serves as a user k The corresponding beam vector, For users kInterference power, The first in the base station beamforming matrix i Each element.

[0009] System total rate The calculation formula is: , K This represents the total number of users.

[0010] The calculation formula for the joint optimization problem of the system is as follows: , , , , RIS reflection matrix The Middle There are diagonal elements, and the total number of diagonal elements is equal to . L .

[0011] The RIS considers non-ideal effects, and the RIS three-dimensional reflection amplitude varies with the phase. The RIS reflection matrix is ​​obtained based on the phase-correlation attenuation coefficient.

[0012] In S3, all commentator networks are isomorphic networks, using independent parameter initialization. During training, they are independently optimized based on the same environmental interaction data, and each commentator network outputs an independent target Q-value for the same state and action pair.

[0013] A RIS-assisted multi-user MISO system beamforming optimization system, used to implement the above-mentioned RIS-assisted multi-user MISO system beamforming optimization method, includes: A base station is equipped with M transmitting antennas to send independent data streams to K single-antenna users. The base station beamforming matrix is ​​configured to achieve active beamforming, where M ≥ K. The RIS contains L programmable passive reflective units and is a non-line-of-sight propagation link with the base station. It is used to perform phase modulation and amplitude attenuation on the incident signal from the base station and reconstruct the wireless propagation environment. The user end consists of K single-antenna users, and the non-line-of-sight propagation link between it and the RIS is used to receive signals only through the reflection link from the base station to the RIS to the user. The controller communicates with the base station and RIS, and stores an executable program. When the controller calls and executes the executable program, it implements the above-mentioned joint beamforming and RIS phase shift optimization method, outputs the base station beamforming matrix and transmits it to the base station, and outputs the RIS reflection matrix and transmits it to the RIS.

[0014] A RIS-assisted multi-user MISO system beamforming optimization device includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the aforementioned RIS-assisted multi-user MISO system beamforming optimization method.

[0015] A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned beamforming optimization method for a RIS-assisted multi-user MISO system.

[0016] The beneficial effects of this invention are as follows: This invention provides a beamforming optimization method for a RIS-assisted multi-user MISO system. First, a RIS-assisted multi-user downlink MISO communication system model is constructed, closely reflecting the actual transmission characteristics of block fading channels. By cascading equivalent channels, the reflection transmission link from the base station to the RI to the user is accurately characterized, precisely capturing the system signal transmission patterns and laying a realistic and reliable model foundation for subsequent joint optimization work. Then, a joint optimization problem is constructed with the goal of maximizing the system's total rate, directly addressing the core issue of multi-user interference by coordinating the optimization of base station beamforming and the RIS reflection matrix. Finally, the high-dimensional non-convex system joint optimization problem is transformed into a Markov decision process in a continuous action space, overcoming the limitations of traditional numerical optimization methods under large-scale RIS configurations and dynamic channel conditions. Overcoming the limitations of high complexity and insufficient real-time performance, this method employs an improved TD3 algorithm. It reduces value assessment bias through independent Q-value estimation from multiple independent reviewer networks, and combines truncated mean aggregation to process the target Q-value. This effectively suppresses Q-value overestimation while avoiding the overly conservative problem caused by the minimum strategy, significantly improving the stability of algorithm training and the robustness of policy learning. This method can accurately output the optimal base station beamforming matrix and RIS reflection matrix, achieving coordinated optimization of active base station beamforming and passive RIS phase shifting. It effectively enhances the user's desired signal, suppresses communication interference between multiple users, and significantly improves the overall communication rate of the system. Furthermore, the algorithm's learning and decision-making characteristics make it more suitable for large-scale RIS and dynamic channel communication applications, exhibiting stronger practicality and scalability. Attached Figure Description

[0017] The solutions and advantages of this application will become clear to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0018] In the attached diagram: Figure 1 This is a schematic diagram of the system model in this embodiment. Figure 2This is a flowchart illustrating the method of this embodiment. Figure 3 The diagram shows the experimental results of the method in this embodiment and existing methods during the training process, as illustrated in the example. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the exemplary embodiments of this application clearer, the technical solutions in the exemplary embodiments of this application are described clearly and completely below. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0020] This embodiment provides a beamforming optimization system for a RIS-assisted multi-user MISO system. (See also...) Figure 1 This is used to implement a beamforming optimization method for a RIS-assisted multi-user MISO system, including: A base station is equipped with M transmitting antennas to send independent data streams to K single-antenna users. The base station beamforming matrix is ​​configured to achieve active beamforming, where M ≥ K. The RIS (Radio Reflector System) comprises L programmable passive reflector units and establishes a non-line-of-sight propagation link with the base station. It is used to perform phase modulation and amplitude attenuation on the incident signal from the base station, reconstructing the wireless propagation environment. Each reflector unit of the RIS is equipped with a phase modulation module and an amplitude attenuation module. The phase modulation module is used to adjust the phase of the reflected signal. The amplitude attenuation module is used to adjust the phase correlation attenuation coefficient of the reflected signal. Each module works in concert to achieve programmable control of the incident signal; The user end consists of K single-antenna users, and the non-line-of-sight propagation link between it and the RIS is used to receive signals only through the reflection link from the base station to the RIS to the user. The controller communicates with the base station and the RIS (Reflection Array), and stores an executable program. When the controller calls and executes the executable program, it implements the joint beamforming and RIS phase shift optimization method as described in claim 1, outputs the base station beamforming matrix and transmits it to the base station, and outputs the RIS reflection matrix and transmits it to the RIS. The controller includes an experience replay pool module and a network training module. The experience replay pool module is used to store sample data of state, action, reward, and next state generated by the interaction between the system and the environment. The network training module is used to implement iterative training and parameter updates of the policy network and commentator network of the improved TD3 algorithm.

[0021] This embodiment provides a beamforming optimization method for a RIS-assisted multi-user MISO system. (See also...) Figure 2 This includes the following operations: S1. Construct a RIS-assisted multi-user downlink MISO communication system model, including a base station with M transmit antennas, a RIS with L reconfigurable reflection units, and K single-antenna users, where M≥K; Under block fading channel conditions, establish the base station RIS channel matrix and the RIS user channel matrix, and combine them with the RIS reflection matrix to obtain the cascaded equivalent channel reflected by the RIS. S2. Based on the cascaded equivalent channel, establish a user received signal model, derive the signal-to-interference-plus-noise ratio of each user, and construct a high-dimensional non-convex joint optimization problem of the system with the goal of maximizing the total system rate. The decision variables of the joint optimization problem include at least the base station beamforming matrix and the RIS reflection matrix. S3. The joint optimization problem of the system is transformed into a Markov decision process in a continuous action space. The state is defined to include channel information and control information, the action includes the base station beamforming matrix and RIS reflection coefficient, and the reward is related to the instantaneous total rate of the system. The improved TD3 algorithm is used to train the Markov decision process and learn the policy, and the base station beamforming matrix and RIS reflection matrix are output as the joint beamforming result. The improved TD3 algorithm includes the use of multiple independent commentator networks, each of which learns independently and provides a target Q-value estimate. The truncated mean of all target Q-values ​​is aggregated to update the target Q-value. The specific steps are detailed below.

[0022] S1. Construct a RIS-assisted multi-user downlink MISO communication system model, including a base station with M transmit antennas, a RIS with L reconfigurable reflection units, and K single-antenna users, where M≥K. Under block fading channel conditions, establish the base station RIS channel matrix and the RIS user channel matrix, and combine them with the RIS reflection matrix to obtain the cascaded equivalent channel reflected by the RIS.

[0023] This embodiment addresses the problem of operating a RIS-assisted multi-user downlink MISO communication system. Due to obstructions, the direct link between the base station (BS) and users suffers severe transmission loss. Therefore, this embodiment ignores the direct link and only considers communication via the reflection link through the RIS. The RIS consists of multiple programmable passive reflection units that, under the control of a controller, apply phase modulation to the incident signal, thereby directionally reflecting the base station signal to different users. Through joint control of the reflected beams, the RIS can reconstruct the propagation environment and improve the performance of multi-user downlink communication.

[0024] In this embodiment, see Figure 1The RIS-assisted multi-user downlink MISO communication system model consists of a base station with M transmit antennas, a RIS with L reconfigurable reflective elements, and K single-antenna users, where M ≥ K. The base station transmits independent data streams to multiple users simultaneously through the RIS, with each user corresponding to an independent data stream.

[0025] Meanwhile, the system in this embodiment operates under block fading channel conditions, meaning the channel remains constant within a single channel coherence interval but varies independently between different intervals. Therefore, to quantify the solution problem of the system in this embodiment, this embodiment establishes a base station RIS channel matrix and a RIS user channel matrix under block fading channel conditions, and combines them with the RIS reflection matrix to obtain the cascaded equivalent channel reflected by RIS.

[0026] The base station RIS channel matrix, i.e., the channel matrix between the base station and the RIS, is as follows: , represents the channel matrix from the base station's n antennas to the 𝐿 reflection units of the RIS. M and represent the total number of antennas and the total number of reflector elements, respectively, and the matrix... The (ℓ, m) The element represents the number of base stations. m root antenna to RIS ℓ The complex channel gain of each reflection unit. Assuming the link operates in a non-line-of-sight (NLoS) propagation environment, its channel coefficients follow an independent and identically distributed complex Gaussian distribution, i.e. It is used to characterize Rayleigh fading properties.

[0027] The RIS user channel matrix, i.e., the channel matrix between the RIS and the users, is represented as follows: ,in K This represents the total number of users. (Matrix) The (ℓ, k) The element represents the number from RIS. ℓ One reflective unit to the user k The complex channel gain. Similarly, this embodiment assumes that the RIS-user link is in a non-line-of-sight propagation environment, satisfying... And with They are independent of each other.

[0028] The RIS employs a single-port, passive reflection, fixed-amplitude model. The reflection behavior of the RIS can be determined by a diagonal matrix. Depiction, among which For the first The equivalent reflection coefficient of a RIS unit. Ideally, the RIS unit only applies phase modulation to the incident signal, i.e., the ideal equivalent reflection coefficient of the RIS reconfigurable reflective unit. , Let be the phase of the reflected signal. To more closely approximate actual hardware characteristics, this embodiment of the RIS considers non-ideal effects; the three-dimensional reflection amplitude of the RIS varies with the phase, and the actual reflection matrix of the RIS is based on the phase-correlation attenuation coefficient. Therefore, the (actual) RIS reflection matrix can be described as It accurately depicts the reflection loss introduced by the actual hardware.

[0029] Therefore, ignoring the direct link between the base station and the user, the equivalent propagation process of the system can be described by the concatenated channel. The concatenated equivalent channel includes the system's equivalent concatenated channel and the user's equivalent concatenated channel. The system's equivalent concatenated channel can be expressed as: , No. k The equivalent concatenated channel for each user is: This equivalent channel characterizes the reconstructive effect of RIS on the wireless propagation environment, facilitating subsequent received signal modeling and performance analysis. Under the block fading assumption, the channel remains constant within a transport block but varies independently between different transport blocks. This embodiment assumes that the base station can obtain the instantaneous channel state information (CSI) of each link for joint beamforming and RIS phase shift optimization.

[0030] S2. Based on the cascaded equivalent channel, establish a user received signal model, derive the signal-to-interference-plus-noise ratio of each user, and construct a high-dimensional non-convex joint optimization problem of the system with the goal of maximizing the total system rate; the decision variables of the joint optimization problem include at least the base station beamforming matrix and the RIS reflection matrix.

[0031] In this embodiment, the base station uses linear precoding (base station beamforming matrix) to simultaneously transmit independent data streams to K single-antenna users in parallel. Let the base station transmit symbol vector be... The symbols satisfy The base station beamforming matrix is ​​represented as follows: , Given the beam vector corresponding to user K, and ignoring the direct link between the base station and the user, the effective propagation process of the system is completed through the reflection path from the base station to the RIS to the user. k The received baseband signal can be expressed as: ,in for The k List, To send to users i Information symbols that satisfy , The additive white Gaussian noise can be further expanded to obtain the structure of "desired signal + multi-user interference + noise": , Therefore, it can be seen that the first k The received signal for each user consists of three parts: the desired signal after RIS reflection, multi-user interference, and additive noise. Multi-user interference, stemming from the non-orthogonality of different user beams in the cascaded channel, is one of the main factors limiting system performance. For ease of subsequent analysis, this embodiment introduces the aforementioned cascaded equivalent channel, thus simplifying the received signal model calculation formula as follows: This calculation formula clearly describes the combined effect of the RIS phase shift matrix and base station beamforming on the desired signal enhancement and multi-user interference suppression, laying the foundation for the subsequent derivation of the user signal-to-interference-plus-noise ratio (SINR) expression and the optimization of the system's total rate.

[0032] For the system model, this embodiment addresses the issue of improving the overall communication performance of the system by jointly designing the base station beamforming matrix and the RIS reflection matrix. Specifically, given channel state information, the system needs to rationally configure the base station's transmit beam and the RIS reflection response to enhance the desired signal while suppressing multi-user interference.

[0033] Therefore, based on the above received signal model, the user... k The desired signal power can be expressed as ,user k Interference power After considering the impact of noise, the user k The formula for calculating the signal-to-interference-plus-noise ratio is as follows: The noise term is normalized based on the number of users. This is the normalized equivalent noise term for multi-user parallel transmission, used to characterize the equivalent noise level in multi-user downlink scenarios.

[0034] With the total system rate As an evaluation indicator, the calculation formula is: This indicator comprehensively reflects the synergistic effect of RIS phase configuration and base station precoding in improving the desired signal and suppressing multi-user interference, and is the core objective of subsequent optimization and learning.

[0035] Therefore, the joint optimization problem of the system model in this embodiment is as follows: , , , , The first one in the RIS reflection matrix There are n elements, and the total number of elements is equal to 1. L , This represents the maximum transmission power of the base station.

[0036] S3. The joint optimization problem of the system is transformed into a Markov decision process in a continuous action space. The state includes channel information and control information, the action includes the base station beamforming matrix and RIS reflection coefficient, and the reward is related to the instantaneous total rate of the system. The improved TD3 algorithm is used to train and learn the policy of the Markov decision process, and the base station beamforming matrix and RIS reflection matrix are output as the joint beamforming result.

[0037] Because the system model in this embodiment has high-dimensional and continuous optimization variables, a strong coupling relationship between the base station beamforming matrix and the RIS phase shift matrix in the channel gain, and high-dimensional non-convex characteristics due to user interference and shared unit transmission power, the joint optimization problem in S2 is also characterized by high dimensionality. Against this backdrop, traditional iterative optimization (AO), semidefinite relaxation (SDR), and fractional programming (FP) methods face problems such as high computational complexity and poor real-time performance in large-scale RIS scenarios. To solve this technical problem, this embodiment transforms the joint optimization problem into a Markov decision process in a continuous action space. An improved TD3 algorithm is used to train and learn the policy for the Markov decision process, effectively reducing value estimation bias while further improving the stability and robustness of the policy learning process. The output base station beamforming matrix and RIS reflection matrix are then used as the joint beamforming result.

[0038] The Markov decision process is modeled as a quintuple. , , , , , These represent the state space, action space, state transition probability, reward function, and discount factor, respectively. Under the block fading assumption, the channel remains constant (or changes slowly) within a time slot / episode. At each step, the agent outputs an action based on the observed state. The environment updates the base station beamforming and RIS phase shift accordingly, and feeds back the reward and the next state, thus forming a closed-loop interaction.

[0039] Define the state space The state includes channel information and control information, specifically at time step. t The state vector observed by the agent at that time is denoted as Used to characterize the instantaneous channel environment, power distribution, and current control strategy information of the system, setting the system state vector. Defined as: , Represents the previous moment (time step) t -1) control actions, including the real and imaginary expansion forms of the base station beamforming matrix and the RIS phase matrix; For time stept Base station transmit power vector, It reflects the energy allocation of each user's corresponding beam; For time step t The equivalent received power vector is calculated based on the equivalent cascaded channel to determine the received signal energy of each user and is used to describe the instantaneous gain of the link. These are the real and imaginary parts of the RIS channel matrix, respectively. These are the real and imaginary parts of the RIS channel matrix, respectively. The real and imaginary parts of the complex matrix are taken respectively. By performing real-imaginary separation processing on the complex channel, it can be used as the input feature of the neural network without losing channel information.

[0040] Define action space The action includes the base station beamforming matrix and RIS reflection coefficient, specifically at time step t The agent outputs a continuous action This is used to jointly optimize active beamforming and RIS passive phase control at the base station. Therefore, the action vector is defined as follows: The action space consists of the base station beamforming matrix and the RIS reflection coefficient. The real and imaginary parts are combined to adapt to the real-valued input and output requirements of deep neural networks. Therefore, the dimension of the action space is: The action space is a high-dimensional continuous space, which is suitable for solving using deep reinforcement learning algorithms based on policy gradients.

[0041] A reward function is defined to characterize the agent's immediate feedback on system performance given a state and action. Based on the previously established signal model, this embodiment uses maximizing the system's total rate of return as the optimization objective of reinforcement learning. The system at time step... t The instant reward is defined as: This reward function is equivalent to the instantaneous total rate of the system, and can simultaneously reflect the combined effect of desired signal enhancement and multi-user interference suppression, which is consistent with the original objective function of the joint beamforming and RIS phase optimization problem.

[0042] In RIS-assisted multi-user MISO downlink scenarios, base stations need to jointly optimize continuous beamforming variables and RIS phase variables to improve overall system performance and rate. This problem is characterized by high variable dimensionality, complex constraints, and a strongly non-convex objective function. It incurs significant computational overhead in dynamic channel environments and large-scale RIS configurations, and is prone to getting trapped in local optima. In this scenario, due to the rapid increase in action dimension with the number of RIS units and the high sensitivity of the reward function to action perturbations, the traditional TD3 algorithm may exhibit overly conservative or slow convergence issues. Therefore, this embodiment improves upon the TD3 framework, proposing an improved TD3 algorithm. This improved TD3 algorithm includes using multiple independent commentator networks, where each commentator independently learns and provides a target Q-value estimate. Then, the truncated mean of all target Q-values ​​is aggregated to update the target Q-value for the current step.

[0043] In the improved TD3 algorithm, all commentator networks are isomorphic, employing independent parameter initialization. During training, they are independently optimized based on the same environmental interaction data, and each commentator network outputs an independent target Q-value for the same state and action pair. Specifically, the state-action value function is no longer approximated by a single or dual commentator network, but is represented by a set containing N independent commentator networks: N is a positive integer greater than 2. All critic networks have the same network structure but are initialized and optimized independently. During training, each critic network learns based on the same environmental interaction data, but due to random initialization and stochastic gradient updates, their value estimates have certain numerical differences. By evaluating the long-term rewards of the same state-action pair from different perspectives using multiple approximation functions, the systematic nature of a single Q-network in high-noise environments can be effectively reduced.

[0044] Furthermore, considering that using all critics in each update may introduce additional computational burden, in order to simultaneously reduce the diversity among critic networks, the improved TD3 algorithm in this embodiment uses a random subset sampling mechanism in the Q-value estimation stage. Specifically, in each update process of the critic network and the policy network, M critics are randomly selected from N target critic networks to form a critic subset. M is a positive integer and M≤N, and is only used for calculating the target Q value.

[0045] After selecting a subset of critics The improved TD3 algorithm does not follow the traditional TD3 strategy of simply taking the minimum Q value. Instead, it introduces a truncated mean aggregation mechanism to achieve a balance between conservatism and estimation bias. Specifically, it targets the set of target Q values ​​for all commentator network outputs, or a subset of commentators. The corresponding set of objective Q-values ​​is sorted, and the smallest first-order objective Q-value is removed to avoid overly conservative value estimation. The average of the remaining objective Q-values ​​is taken to obtain the truncated mean objective Q-value, which is used for policy updates to update the objective Q-value of the current step. Therefore, the temporal difference objective of the critic network is constructed as follows: ,in As a discount factor, By truncating the mean target Q value, this approach can effectively avoid the impact of some commentators' underestimation of the Q value on the overall Q value aggregation result. The truncated mean aggregation strategy can make the Q value update more stable and avoid the influence of extreme Q values ​​on the learning process. In this way, the algorithm can avoid the problem of overly conservative or slow convergence when facing high-dimensional action spaces, thus accelerating the learning process and improving the convergence speed.

[0046] In each parameter update, the improved TD3 algorithm employs a shared objective but independent optimization training method for all commentator networks. That is, all commentator networks use the same target Q-value as the supervision signal, but their parameter update processes are independent of each other. i Loss function for each commentator as follows: , For the first i A network of critics estimates the target Q-value for the current state and action pair. For the time-difference objective, This is a mathematical expectation operation.

[0047] This training method effectively maintains the diversity of the critic set in terms of value estimation while ensuring consistency of the objective. To avoid frequent updates of the policy network interfering with the stability of the value function, the improved TD3 algorithm uses a random subset of critics as the basis for value evaluation during the policy update phase. N critic networks are randomly selected from the critic network set, and the average of their Q-values ​​is used as the policy optimization objective. The policy loss function... The calculation formula is as follows: , For the policy network in the current state s The continuous actions output below.

[0048] This consistency design ensures that the critic network's optimization objective aligns with the objective Q-estimation method, thereby reducing bias propagation during policy gradient updates. After the agent update is complete, a soft update is performed on the target network parameters of both the agent and all critic networks. , , These are the parameters for the policy network and the actor target network, respectively. , For the first i A network of critics, the first i Parameters of the Critic target network, These are soft update coefficients. This mechanism ensures the target network evolves smoothly over time, thereby further enhancing the stability of the training process.

[0049] To verify the effectiveness of the method in this embodiment, the following experiment was conducted.

[0050] Parameter Settings. The experiment was run on a computer equipped with an NVIDIA GPU, using CUDA to accelerate the training process. The experiment employed the PyTorch deep learning framework and the Gym reinforcement learning environment to train and simulate the algorithm, and to train and evaluate the system model. In this experiment, the performance of the method in this embodiment was evaluated in a RIS-assisted multi-user MISO system through training. To ensure the algorithm could learn stably and optimize system performance in a multi-user environment, each experiment was trained for 104 steps to comprehensively evaluate the superiority of the method in this embodiment. The experiment included environmental parameters and DRL algorithm parameters. Environmental parameters were mainly used to simulate the system's communication configuration and channel conditions to ensure the experiment reflected the challenges in real-world applications; while the DRL algorithm parameters included key hyperparameters such as the learning rate and batch size that control the deep reinforcement learning process. Specific parameter settings are shown in Table 1.

[0051] Table 1. Parameter Settings .

[0052] To more comprehensively analyze and evaluate the performance of the method in this embodiment in a RIS-assisted multi-user MISO system, the following comparative experiments were designed to demonstrate the advantages of the method in different aspects.

[0053] The training process and reward value of the method in this embodiment were demonstrated in the experiment as follows: Figure 3 As shown, the method of this embodiment (corresponding to) is illustrated. Figure 3 The red curve in the image represents the Proposed Algorithm, corresponding to the improved TD3 algorithm in this embodiment, and DDPG (corresponding to...). Figure 3 The orange curve DDPG in the image) and the TD3 algorithm (corresponding to Figure 3The blue curve (TD3) in the figure illustrates the change in reward with the number of steps during training, reflecting the performance of different algorithms in the RIS-assisted multi-user MISO system. As can be seen from the figure, the method in this embodiment (red curve) exhibits a stable reward growth curve. During training, the reward value of this embodiment continuously increases with relatively small fluctuations, indicating that this embodiment can converge quickly and maintain stable performance during training. Compared with the method in this embodiment, the DDPG algorithm (orange curve) and the TD3 algorithm (blue curve) show greater volatility. DDPG converges more slowly, especially in the initial stage, with a slow increase in reward, and exhibits significant fluctuations in the later stages of training. The TD3 algorithm converges faster, but also suffers from large reward fluctuations, especially near the end of training, where the fluctuations in reward increase significantly.

[0054] These results demonstrate that the proposed method exhibits significant advantages in RIS phase control and beamforming optimization, maintaining stable performance and fast convergence speed even under uncertainties such as multi-user interference and channel noise. Through a multi-commentator mechanism and truncated mean, the proposed method successfully avoids overestimation of the Q-value, enabling the algorithm to learn stably and efficiently, thereby improving the overall system rate.

[0055] This embodiment also provides a RIS-assisted multi-user MISO system beamforming optimization device, including a processor and a memory, wherein the processor executes the computer program stored in the memory to implement the above-described RIS-assisted multi-user MISO system beamforming optimization method.

[0056] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described RIS-assisted multi-user MISO system beamforming optimization method.

[0057] This embodiment provides a beamforming optimization method for a RIS-assisted multi-user MISO system. First, a RIS-assisted multi-user downlink MISO communication system model is constructed, closely reflecting the actual transmission characteristics of block fading channels. By using cascaded equivalent channels, the reflection transmission link from the base station to the RI to the user is accurately characterized, precisely capturing the system signal transmission patterns and laying a realistic and reliable model foundation for subsequent joint optimization work. Then, a joint optimization problem is constructed with the goal of maximizing the system's total rate, directly addressing the core issue of multi-user interference by coordinating the optimization of base station beamforming and the RIS reflection matrix. Finally, the high-dimensional non-convex system joint optimization problem is transformed into a Markov decision process in a continuous action space, overcoming the limitations of traditional numerical optimization methods under large-scale RIS configurations and dynamic channel conditions. Overcoming the limitations of high computational complexity and insufficient real-time performance, this method employs an improved TD3 algorithm. It reduces value assessment bias through independent Q-value estimation from multiple independent reviewer networks, and combines truncated mean aggregation to process the target Q-value. This effectively suppresses Q-value overestimation while avoiding the overly conservative problem caused by the minimum strategy, significantly improving the stability of algorithm training and the robustness of policy learning. This method can accurately output the optimal base station beamforming matrix and RIS reflection matrix, achieving coordinated optimization of active base station beamforming and passive RIS phase shifting. This effectively enhances the user's desired signal, suppresses communication interference between multiple users, and significantly improves the overall communication rate of the system. Furthermore, the algorithm's learning and decision-making characteristics make it more suitable for large-scale RIS and dynamic channel communication applications, exhibiting stronger practicality and scalability.

[0058] While exemplary embodiments of the invention have been described herein, many other variations or modifications conforming to the principles of the invention can be directly determined or derived from the disclosure of this invention without departing from its spirit and scope. Therefore, the scope of the invention should be understood and recognized to cover all such other variations or modifications.

Claims

1. A beamforming optimization method for a RIS-assisted multi-user MISO system, characterized in that, This includes the following operations: S1. Construct a RIS-assisted multi-user downlink MISO communication system model, including a base station with M transmit antennas, a RIS with L reconfigurable reflective elements, and K single-antenna users, where M≥K; Under block fading channel conditions, the base station RIS channel matrix and RIS user channel matrix are established, and combined with the RIS reflection matrix, the cascaded equivalent channel reflected by RIS is obtained. S2. Based on the cascaded equivalent channel, establish a user received signal model, derive the signal-to-interference-plus-noise ratio of each user, and construct a high-dimensional non-convex system joint optimization problem with the goal of maximizing the total system rate. The decision variables for the joint optimization problem include at least the base station beamforming matrix and the RIS reflection matrix; S3. The joint optimization problem of the system is transformed into a Markov decision process in a continuous action space. The state includes channel information and control information, the action includes the base station beamforming matrix and RIS reflection coefficient, and the reward is related to the instantaneous total rate of the system. An improved TD3 algorithm is used to train the Markov decision process and learn the policy, and output the base station beamforming matrix and RIS reflection matrix as the joint beamforming result. The improved TD3 algorithm includes the use of multiple independent commentator networks, each of which learns independently and provides a target Q-value estimate. The truncated mean of all target Q-values ​​is then aggregated to update the target Q-value.

2. The beamforming optimization method for a RIS-assisted multi-user MISO system according to claim 1, characterized in that, In S3, the truncated mean aggregation operation is as follows: sort the target Q-value sets of all commentator network outputs, remove the smallest first-order target Q-values, and take the mean of the remaining target Q-values ​​to obtain the truncated mean target Q-values.

3. The beamforming optimization method for a RIS-assisted multi-user MISO system according to claim 1, characterized in that, In the Q-value estimation stage of S3, a random subset sampling mechanism is used. Specifically, during each update of the critic network and the policy network, M critics are randomly selected from the N target critic networks to form a critic subset, where M is a positive integer greater than 2 and M≤N, and this subset is used for target Q-value estimation.

4. The beamforming optimization method for a RIS-assisted multi-user MISO system according to claim 1, characterized in that, In S2, The formula for calculating the signal-to-interference-plus-noise ratio (SIR) for a user is as follows: ， ， ， For the first k Signal-to-interference-to-noise ratio for each user For users k The expected signal power For the normalized equivalent noise term under multi-user parallel transmission, RIS user channel matrix The element in the k-th column, For the RIS reflection matrix, For the base station RIS channel matrix, Base station beamforming matrix The Middle k Each element also serves as a user k The corresponding beam vector, For users k Interference power, The first in the base station beamforming matrix i One element; System total rate The calculation formula is: ， K This represents the total number of users.

5. The beamforming optimization method for a RIS-assisted multi-user MISO system according to claim 4, characterized in that, The calculation formula for the joint optimization problem of the system is as follows: ， ， ， ， RIS reflection matrix The Middle There are diagonal elements, and the total number of diagonal elements is equal to . L .

6. The beamforming optimization method for a RIS-assisted multi-user MISO system according to claim 1, characterized in that, The RIS considers non-ideal effects, and the RIS three-dimensional reflection amplitude varies with the phase. The RIS reflection matrix is ​​obtained based on the phase-correlation attenuation coefficient.

7. The beamforming optimization method for a RIS-assisted multi-user MISO system according to claim 1, characterized in that, In S3, all commentator networks are isomorphic networks, using independent parameter initialization. During training, they are independently optimized based on the same environmental interaction data, and each commentator network outputs an independent target Q-value for the same state and action pair.

8. A RIS-assisted multi-user MISO system beamforming optimization system, used to implement the RIS-assisted multi-user MISO system beamforming optimization method of claim 1, characterized in that, include: A base station is equipped with M transmitting antennas to send independent data streams to K single-antenna users. The base station beamforming matrix is ​​configured to achieve active beamforming, where M ≥ K. The RIS contains L programmable passive reflective units and is a non-line-of-sight propagation link with the base station. It is used to perform phase modulation and amplitude attenuation on the incident signal from the base station and reconstruct the wireless propagation environment. The user end consists of K single-antenna users, and the non-line-of-sight propagation link between it and the RIS is used to receive signals only through the reflection link from the base station to the RIS to the user. The controller communicates with the base station and the RIS, and stores an executable program. When the controller calls and executes the executable program, it implements the joint beamforming and RIS phase shift optimization method as described in claim 1, outputs the base station beamforming matrix and transmits it to the base station, and outputs the RIS reflection matrix and transmits it to the RIS.

9. A beamforming optimization device for a RIS-assisted multi-user MISO system, characterized in that, It includes a processor and a memory, wherein the processor implements the RIS-assisted multi-user MISO system beamforming optimization method as described in any one of claims 1-7 when executing a computer program stored in the memory.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the RIS-assisted multi-user MISO system beamforming optimization method as described in any one of claims 1-7.

Images