Multi-RIS-assisted wireless network coverage optimization method and device

Abstract

The invention relates to the technical field of wireless communication, and provides a multi-RIS-assisted wireless network coverage optimization method and device, and the method comprises the steps: building a channel model of a multi-RIS-assisted wireless communication network; constructing a target reward function based on a channel model of a multi-RIS-assisted wireless communication network; constructing an optimization problem by taking a maximized target reward function as a target, taking a deployment position and a phase angle of each RIS as optimization variables and taking a deployment range and a phase angle range of each RIS as constraint conditions; and carrying out iterative solution on the optimization problem by adopting a depth deterministic strategy gradient algorithm to obtain the deployment position and phase angle of each optimized RIS so as to carry out network coverage optimization on the wireless communication network. According to the method, the deployment positions and phase angles of a plurality of RISs are subjected to collaborative optimization, and the wireless network coverage rate optimization effect is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a multi-RIS-assisted wireless network coverage optimization method and device. Background Art

[0002] In recent years, as wireless data traffic has increased dramatically, people's requirements for communication service quality have also increased simultaneously, which has stimulated a strong demand for future wireless communication speed, quality and security. To achieve this vision, reconfigurable intelligent surface (RIS) has become an emerging paradigm for wireless communication, which has significant advantages over traditional solutions. RIS can provide line-of-sight (LOS) links to users in blocked areas through reflection to improve the coverage and capacity of traditional wireless networks.

[0003] In order to further improve the wireless network coverage, RIS is generally used to assist in optimizing the wireless network coverage. However, existing research mainly focuses on optimizing the phase shift of RIS, that is, optimizing network coverage by adjusting the phase angle of RIS, and current research mainly focuses on optimizing the phase shift of a single RIS to enhance network coverage. However, this optimization scheme that only optimizes the RIS phase shift has certain limitations, and the optimization effect of the final network coverage is ineffective. Summary of the invention

[0004] Based on this, an object of the present invention is to provide a multi-RIS-assisted wireless network coverage optimization method and device to solve at least one technical problem in the background technology.

[0005] A multi-RIS-assisted wireless network coverage optimization method according to an embodiment of the present invention is applied to a multi-RIS-assisted wireless communication network, and the method comprises the following steps: Establishing a channel model of the multi-RIS-assisted wireless communication network according to the system architecture of the multi-RIS-assisted wireless communication network; constructing a target reward function based on a channel model of the multi-RIS-assisted wireless communication network; The optimization problem is constructed with the goal of maximizing the target reward function, the deployment position and phase angle of each RIS as optimization variables, and the deployment range and phase angle range of each RIS as constraints; A preset deep deterministic policy gradient algorithm is used to iteratively solve the optimization problem to obtain the optimized deployment position and phase angle of each RIS, so as to optimize the network coverage of the multi-RIS-assisted wireless communication network.

[0006] In addition, the multi-RIS-assisted wireless network coverage optimization method according to the above embodiment of the present invention may also have the following additional technical features: Furthermore, the channel model of the multi-RIS-assisted wireless communication network includes: Base station to grid point and Channel gain haha, yes , Channel gain from base station to RIS Ha,r , and RIS to grid points and Channel gain yes, yes , specifically defined as:

[0007] in, and ∈{ s 1, s 2,……, sK},in K is the total number of grids within the service range of the base station, ε 1. ε 2 and ε 3 represent haha, yes , Ha,r and yes, yes The Rician factor of the channel link, L 1. L 2 and L 3 represent haha, yes , Ha,r and yes, yes The path loss is Respectively haha, yes , Ha,r and yes, yes The LoS component, Respectively haha, yes , Ha,r and yes, yes The NLoS component of

[0008] Furthermore, the target reward function is:

[0009] in, Rr ( t ) represents r The target reward function of a RIS is M Represents the number of RIS.

[0010] Further, the multi-RIS-assisted wireless communication network includes a RIS for primary base station assistance and a RIS for interfering base station assistance; Among them, the objective reward function of RIS for primary base station assistance is: R( t )= R ( t ) RSRP + R ( t ) SINR Among them, the objective reward function of the RIS used to interfere with the base station assisted RIS is: R ( t ) = R ( t ) SINR in,

[0011] in,

[0012] In the formula, R ( t ) RSRP represents the target reward function for the reference signal received power, R ( t ) SINR The target reward function represents the signal-to-interference-plus-noise ratio, Representing grid points and The target reward function for the reference signal received power at , Representing grid points and The target reward function for the signal-to-interference-plus-noise ratio at , RSRPsi Representing grid points and The reference signal received power at SINRsi Representing grid points and The signal to interference plus noise ratio at THRSRP and THSINR is the corresponding threshold.

[0013] Further, wherein:

[0014] In the formula, Represents the main base station to the grid point and The channel gain matrix at Represents the interfering base station to the grid point and The channel gain matrix at Represents the RIS to the grid point assisted by the main base station and The channel gain matrix, Represents the RIS to the grid point assisted by the interfering base station and The channel gain matrix, represents the channel gain matrix from the primary base station to the primary base station-assisted RIS, represents the channel gain matrix from the interfering base station to the interfering base station assisted RIS, and represents the diagonal matrix of the primary base station and the interfering base station, x 1 and x 2 represent the amount of user data to be transmitted by the primary base station and the interfering base station respectively, v 1 represents the dynamic noise caused by the RIS assisted by the main base station, n Represents environmental noise.

[0015] Furthermore, the step of using a preset deep deterministic policy gradient algorithm to iteratively solve the optimization problem includes: Every preset time t , control each RIS in the current state Select actions under the environment and the current policy network and execute; Get the actions being performed by each RIS Rewards returned from the later environment And observe the next state , to obtain multiple sets of training samples , and multiple sets of training samples Stored in the experience replay buffer; A batch of training samples is randomly extracted from the experience replay buffer, and the Q value of each RIS in the current policy network is calculated. The parameters of the policy network are iteratively updated by minimizing the mean square error until the network converges, and the final network parameters are output to obtain the optimized deployment position and phase angle of each RIS.

[0016] Furthermore, for the RIS for primary base station assistance, the step of updating the parameters of the policy network includes: Adopting the preset policy asynchronous update rules to asynchronously update the deployment position and phase angle of the RIS of the policy network; Among them, the strategy asynchronous update rule is to make the RIS deployment position update frequency slower than the phase angle update frequency. Specifically, the phase angle is optimally solved when the RIS deployment position is fixed, and then the RIS deployment position is updated, and this is iterated until convergence.

[0017] A multi-RIS-assisted wireless network coverage optimization device according to an embodiment of the present invention includes: A model building module, used to build a channel model of the multi-RIS-assisted wireless communication network according to the system architecture of the multi-RIS-assisted wireless communication network; A function construction module, for constructing a target reward function based on a channel model of the multi-RIS-assisted wireless communication network; The problem construction module is used to construct an optimization problem with the goal of maximizing the target reward function, the deployment position and phase angle of each RIS as optimization variables, and the deployment range and phase angle range of each RIS as constraints; The coverage optimization module is used to iteratively solve the optimization problem using a preset deep deterministic policy gradient algorithm to obtain the optimized deployment position and phase angle of each RIS, so as to optimize the network coverage of the multi-RIS-assisted wireless communication network.

[0018] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, which, when executed by a processor, implements the multi-RIS-assisted wireless network coverage optimization method as described above.

[0019] A multi-RIS-assisted wireless network coverage optimization device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the multi-RIS-assisted wireless network coverage optimization method as described above is implemented.

[0020] Compared with the existing technology: by taking the deployment position and phase angle of each RIS as the optimization variable, on the one hand, the deployment position and phase angle of RIS are considered, on the other hand, the environmental coordination between multiple RIS is also considered, so that the deployment positions and phase angles of multiple RIS are collaboratively optimized, and a deep deterministic policy gradient algorithm that can realize multi-task and multi-agent collaboration is used to solve the optimization problem, which greatly improves the optimization effect of wireless network coverage. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of the multi-RIS-assisted wireless network coverage optimization method in the first embodiment of the present invention; Figure 2 An example diagram of a system architecture of a multi-RIS-assisted wireless communication network provided in an embodiment of the present invention; Figure 3 A schematic diagram of the overall algorithm framework provided in an embodiment of the present invention; Figure 4 A schematic diagram of a logical framework of an asynchronous update provided in an embodiment of the present invention; Figure 5 It is a structural diagram of a multi-RIS-assisted wireless network coverage optimization device in the third embodiment of the present invention; Figure 6 It is a schematic diagram of the structure of a multi-RIS-assisted wireless network coverage optimization device in a fourth embodiment of the present invention.

[0022] The following specific implementations will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0023] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0024] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0026] At present, when optimizing the wireless network coverage of the RIS-assisted system, the focus is mainly on optimizing the phase shift of RIS, that is, optimizing the network coverage by adjusting the phase angle of RIS. There are mainly the following methods: 1) Optimization-based methods: Optimization algorithms (such as convex optimization, nonlinear programming, etc.) are used to optimize the phase configuration of RIS to achieve network coverage optimization. Such a solution usually cannot meet the real-time rate requirements in large-scale networks. In addition, it reaches a solution after multiple iterations, and its computational complexity cannot be well expanded. 2) Game theory-based methods: The problem is formulated as a non-cooperative game / optimization problem between network entities. Network entities (i.e., players) try to solve the game in a greedy and non-cooperative manner to achieve their goals. 3) Reinforcement learning algorithms (such as Q learning, policy gradient, etc.) are used to learn the optimal RIS phase configuration strategy to optimize network coverage.

[0027] However, no matter which research method is used, since it only considers the optimization of the phase shift of RIS and mainly focuses on optimizing the phase shift of a single RIS, there are certain limitations, and the optimization effect of the network coverage is only effective.

[0028] To this end, an embodiment of the present invention provides a multi-RIS-assisted wireless network coverage optimization method, which collaboratively optimizes the deployment positions and phase angles of multiple RISs and adopts a deep deterministic policy gradient algorithm that can realize multi-task and multi-agent collaboration to solve the optimization problem, thereby greatly improving the optimization effect of wireless network coverage.

[0029] In order to more clearly illustrate a multi-RIS-assisted wireless network coverage optimization method in an embodiment of the present invention, a specific embodiment will be given below.

[0030] Embodiment 1 See also Figure 1 , shown is a multi-RIS-assisted wireless network coverage optimization method in the first embodiment of the present invention, which can be specifically implemented by software and / or hardware. The method specifically includes steps S01 to S04.

[0031] Step S01 : establishing a channel model of the multi-RIS-assisted wireless communication network according to the system architecture of the multi-RIS-assisted wireless communication network.

[0032] As an optional implementation, Figure 2 The system architecture of a multi-RIS-assisted wireless communication network is shown, specifically a dual-RIS-assisted network, which consists of a main base station with M antennas and an interfering base station, and each base station is equipped with N reflective elements near the RIS-assisted communication. In the embodiment of the present invention, the Active RIS near the main base station (i.e., the RIS used for main base station assistance) enhances network coverage by changing phase shift and moving deployment within a certain range, and the Passive RIS near the interfering base station (i.e., the RIS used for interfering base station assistance) changes phase shift at a fixed position to suppress interference, thereby enhancing network coverage.

[0033] Specifically, the coverage area of ​​the primary base station is defined as r The area is discretized into square regions with side length Rց There are many square grids, the total number of grids is K =[ Rs / Rց ]2, the center point of each grid acts as a sample point, which approximately represents the signal quality within the grid. The coverage is approximately estimated as:

[0034] in, It is a set of grid points whose signal quality meets the requirements.

[0035] Grid Points and The received signal yes can be written as:

[0036] in, Indicates the distance from base station a to the grid point and The channel gain vector, represents the channel gain matrix from base station a to RIS, Indicates RIS to grid point and The channel gain vector, is the noise introduced by Active RIS. The phase shift matrix of RIS is composed of the diagonal matrix Given, where each reflection unit n in ActiveRIS defines the reflection coefficient αn =100, coefficient α of each reflection unit of Passive RIS n =1, phase shift of RIS θn ∈[0,2π]. Considering that the two base stations are far away, secondary reflection and interference between the base station and the mobile RIS link are not considered.

[0037] Specifically, the channel model of the multi-RIS-assisted wireless communication network includes: Base station to grid point and Channel gain haha, yes , Channel gain from base station to RIS Ha,r , and RIS to grid points and Channel gain yes, yes , specifically defined as:

[0038] in, and ∈{ s 1, s 2,……, sK},in K is the total number of grids within the service range of the base station, ε 1. ε 2 and ε 3 represent haha, yes , Ha,r and yes, yes The Rician factor of the channel link, L 1. L 2 and L 3 represent haha, yes , Ha,r and yes, yes The path loss is Respectively haha, yes , Ha,r and yes, yes The LoS component, Respectively haha, yes , Ha,r and yes, yes The NLoS component of

[0039] In practice, when the Rician factor is equal to ∞, the Rician fading channel takes the form of a LoS channel. In contrast, when the factor is equal to 0, the channel takes the form of a NLoS channel.

[0040] The NLoS component of the channel is given by For the LoS component, it is given by two types of uniform rectangular arrays (URAS): 1) The planar antenna at the base station has a size of U rows and V columns, M = U × V. 2) The planar reflector at the RIS has a size of P rows and Q columns, N = P × Q. The LoS component is calculated as follows:

[0041] in, They represent the departure azimuth and elevation angle from the base station to the grid point, They represent the arrival azimuth and elevation angle from the base station to the RIS, They represent the departure azimuth and elevation angle from the base station to the RIS, are the departure azimuth and elevation angles from the RIS to the grid point, respectively. represents the receive array response vector arriving at RIS, They represent the transmission array response vectors from the base station to the RIS, the base station to the grid point, and the RIS to the grid point, respectively. The response array is calculated as follows:

[0042] in:

[0043] Where λ is the wavelength and d is the distance between the two elements.

[0044] Step S02: constructing a target reward function based on the channel model of the multi-RIS-assisted wireless communication network.

[0045] Specifically, the target reward function is:

[0046] in, Rr ( t ) represents r The target reward function of a RIS is M Represents the number of RIS.

[0047] In a specific implementation, the multi-RIS-assisted wireless communication network includes a RIS for primary base station assistance and a RIS for interference base station assistance; Among them, the objective reward function of RIS for primary base station assistance is: R ( t )= R ( t ) RSRP + R ( t ) SINR Among them, the objective reward function of the RIS used to interfere with the base station assisted RIS is: R ( t ) = R ( t ) SINR in,

[0048] in,

[0049] In the formula, R ( t ) RSRP represents the target reward function for the reference signal received power, R ( t ) SINR The target reward function represents the signal-to-interference-plus-noise ratio, Representing grid points and The target reward function for the reference signal received power at , Representing grid points and The target reward function for the signal-to-interference-plus-noise ratio at , RSRPsi Representing grid points and The reference signal received power at SINRsi Representing grid points and The signal to interference plus noise ratio at THRSRP and THSINR is the corresponding threshold.

[0050] in:

[0051] In the formula, Represents the main base station to the grid point and The channel gain matrix at Represents the interfering base station to the grid point and The channel gain matrix at Represents the RIS to the grid point assisted by the main base station and The channel gain matrix, Represents the RIS to the grid point assisted by the interfering base station and The channel gain matrix, represents the channel gain matrix from the primary base station to the primary base station-assisted RIS, represents the channel gain matrix from the interfering base station to the interfering base station assisted RIS, and represents the diagonal matrix of the primary base station and the interfering base station, x 1 and x 2 represents the amount of user data to be transmitted by the primary base station and the interfering base station, including the data information transmitted by the primary base station and the interfering base station to their respective associated users. v 1 represents the dynamic noise caused by the RIS assisted by the primary base station, which is a complex Gaussian white noise with a mean of 0 and a variance of σ2. n represents environmental noise, specifically additive white Gaussian noise (AWGN) with a mean of 0 and a variance of σ2.

[0052] In order to improve the quality of service, a basic problem is to maximize the grid coverage ratio. and The coverage index can be defined as:

[0053] That is, only when RSRPsi and SINRsi Both are above the threshold THRSRP and THSINR When , the grid point and is covered. Then the coverage rate is calculated as:

[0054] Step S03, constructing an optimization problem with the goal of maximizing the target reward function, the deployment position and phase angle of each RIS as optimization variables, and the deployment range and phase angle range of each RIS as constraints.

[0055] Specifically, the deployment range and phase angle range constraints of RIS are as follows:

[0056] is the three-dimensional coordinate of RIS, constraint (a) is the RIS phase angle constraint, and constraint (b) limits the RIS to move and deploy in a given space S to adapt to the actual situation and avoid infinite deployment. Specifically, RIS is constant in the z-axis height and constrained in the x-axis and y-axis ranges.

[0057] Step S04, using a preset deep deterministic policy gradient algorithm to iteratively solve the optimization problem, and obtaining the optimized deployment position and phase angle of each RIS, so as to optimize the network coverage of the multi-RIS-assisted wireless communication network.

[0058] Among them, the Deep Deterministic Policy Gradient (DDPG) algorithm is a reinforcement learning algorithm specifically used for continuous action space, which adopts the actor-critic architecture to learn strategies. The actor network is responsible for generating deterministic actions under a given state, while the critic network evaluates the value of these actions and outputs Q values. By using the experience replay buffer to store and randomly sample past experience, DDPG reduces the correlation between samples, and uses the target network for soft updates to improve the stability of the training process. The policy gradient updates the actor network through the feedback of the critic network, so that the actions generated by the strategy can maximize their estimated value. DDPG combines the advantages of policy optimization and value function approximation, and is suitable for realizing efficient policy learning in complex continuous control tasks. The embodiment of the present invention uses the DDPG algorithm to solve the optimization problem of multi-task multi-agent collaboration constructed in step S03, which can quickly and accurately seek the global optimum and improve the optimization efficiency.

[0059] by Figure 2 Taking the dual RIS auxiliary network shown in the figure as an example, the embodiment of the present invention uses the RIS next to the main base station and the interfering base station as the intelligent agent agent1 and the intelligent agent agent2 respectively. The states of the two intelligent agents in the environment at time t are: , perform the action Task rewards can be obtained from the environment, and the collaborative agents have a common goal: to maximize the objective function within the constraints, that is, the coverage of the entire system. However, considering that the two collaborative agents can be decomposed into different tasks, the present invention further decomposes the total reward (that is, the objective function). The objective function is decomposed into two parts: R ( t )= R ( t ) RSRP + R ( t ) SINR

[0060] R ( t ) RSRP and R ( t ) SINRAs two tasks, Task1 and Task2, the overall reward function is decomposed into sub-reward functions, so that the corresponding sub-objective functions can be optimized separately. For agent1, it needs to complete the two tasks, Task1 and Task2, while for agent2, its optimization is only related to SINR task Task2, so it only needs to complete Task2 requirements. The whole algorithm design is as follows Figure 3 shown.

[0061] In summary, the multi-RIS-assisted wireless network coverage optimization method in the above embodiment of the present invention takes the deployment position and phase angle of each RIS as the optimization variable, and considers the deployment position and phase angle of the RIS on the one hand, and the environmental coordination between multiple RIS on the other hand, so as to collaboratively optimize the deployment positions and phase angles of multiple RIS, and adopts a deep deterministic policy gradient algorithm that can realize multi-task and multi-agent collaboration to solve the optimization problem, thereby greatly improving the optimization effect of wireless network coverage.

[0062] Embodiment 2 The second embodiment of the present invention also provides a multi-RIS-assisted wireless network coverage optimization method, which can be implemented by software and / or hardware. The multi-RIS-assisted wireless network coverage optimization method in this embodiment is different from the multi-RIS-assisted wireless network coverage optimization method in the first embodiment in that: The steps of using a preset deep deterministic policy gradient algorithm to iteratively solve the optimization problem include: Every preset time t , control each RIS in the current state Select actions under the environment and the current policy network and execute; Get the actions being performed by each RIS Rewards returned from the later environment And observe the next state , to obtain multiple sets of training samples , and multiple sets of training samples Stored in the experience replay buffer; A batch of training samples is randomly extracted from the experience replay buffer, and the Q value of each RIS in the current policy network is calculated. The parameters of the policy network are iteratively updated by minimizing the mean square error until the network converges, and the final network parameters are output to obtain the optimized deployment position and phase angle of each RIS.

[0063] Specifically, the DDPG algorithm process is divided into three main stages: data sampling stage, strategy evaluation stage, and strategy update stage.

[0064] (1) Data sampling stage: In the data sampling stage, multiple agents interact with the environment and generate a batch of experience data. This data includes the actions taken by each agent in different states, the rewards obtained, and the next state starting from the initial state. The specific process of this stage is as follows: 1. Initialize the environment and the agent’s observation state (For each agent r ).

[0065] 2. For each moment Do the following: First, each agent r Select actions based on the current policy network Executes the selected action.

[0066] Then, according to the corresponding objective function R ( t ), the environment returns a reward And observe the next state (For each agent r ).

[0067] Preferably, each agent's tuple Stored in the experience replay buffer middle.

[0068] 3. End of round: After a certain number of steps are completed, the environment is reset to the initial state and ready for the next round. The agent acquires data by interacting with the environment. Each action performed represents an interaction. After a limited number of interactions (steps) are completed, the environment is reset to the initial state. This entire process is called an episode. Reinforcement learning algorithms collect data from multiple rounds to train the agent.

[0069] (2) Strategy evaluation stage: The goal of the strategy evaluation stage is to evaluate the performance of each agent by calculating the Q value of each agent under the current strategy. Each agent has its own Q network and target Q network. In the multi-task collaborative agent of the embodiment of the present invention, in the setting of the multi-task collaborative agent, each task corresponds to a separate critic network. This setting can help each task independently evaluate the value of the action according to its own specific goal. Agent 1 has two critic grids and two target critic networks, which correspond to the evaluation of the two tasks of agent 1 respectively. Agent 2 has a critic network and a target critic network. Taking the evaluation of agent 1 network as an example: First, the experience replay buffer from each task Randomly select a batch of samples from ; Then, for each sample of each task j : Calculate the target Q value and the Q value of the current Q network; Finally, the mean square error is minimized based on the calculated Q values ​​and used to update the parameters of the Q network for each task.

[0070] (3) Strategy update phase: In the strategy update phase, the critic network independent of each task is used to update the corresponding strategy network parameters. The details are as follows: 1. First calculate the policy gradient and use gradient ascent to update the parameters of the multi-task common policy network 2. Update the target Q network parameters for each task and update the target policy network.

[0071] In addition, before the algorithm is solved, it is necessary to model the actual problem. The most critical elements in the reinforcement learning of the DDPG algorithm include action, state, and reward. Figure 2 Taking the dual RIS auxiliary network shown as an example, these three elements need to be modeled specifically.

[0072] First, for agent 1 (i.e., RIS for primary base station assistance), the modeling is as follows: State: It is defined as the environmental state at time slot t, and each state consists of two parts: the position of the RIS next to the main base station at the previous moment and the channel gain from the main base station to each grid point.

[0073]

[0074] is the position information at the previous moment, and H is the channel gain information.

[0075] Action: At each time slot t, agent 1 takes action based on the state (observation) of the environment feedback. Taking actions based on the policy network , which includes the RIS moving direction and the RIS phase shift angle at time t. The specific form is as follows,

[0076] Reward: The reward function is the objective function. The reward function design plays a key role in the learning process and is usually used to evaluate the quality of the learned decision-making strategy. In the framework of the present invention, the overall reward signal of each agent is decomposed into multiple parts according to the subtasks. Then the proposed algorithm learns multiple value functions with decomposed reward signals and updates the strategy with the sum of the distributed value functions. The specific agent is mainly divided into two tasks: the total RSRP reward and the SINR reward. For agent 1, the reward function can be set as: R ( t )= R ( t ) RSRP + R ( t ) SINR For agent 2 (i.e., RIS used to assist the interfering base station), the modeling is as follows: State: It is defined as the environmental state at time slot t, where the state is: the channel gain from the interfering base station to each grid point.

[0077]

[0078] H is the channel gain information.

[0079] Action: At each time slot t, agent 2 takes action based on the state (observation) of the environment feedback. Taking actions based on the policy network , where includes the phase shift angle of the RIS next to the interfering base station. The specific form is as follows,

[0080] Reward: The overall reward signal of each agent is decomposed into multiple parts according to the subtasks. Then the proposed algorithm learns multiple value functions with decomposed reward signals and updates the strategy with the sum of the step-by-step value functions. The specific agent is mainly divided into two tasks: the total RSRP reward and the SINR reward. For agent 2, the reward function can be set as: R ( t ) = R ( t ) SINR In addition, in some optional embodiments, further, for the RIS for primary base station assistance, the step of updating the parameters of the policy network includes: Adopting the preset policy asynchronous update rules to asynchronously update the deployment position and phase angle of the RIS of the policy network; Among them, the strategy asynchronous update rule is to make the RIS deployment position update frequency slower than the phase angle update frequency. Specifically, the phase angle is optimally solved when the RIS deployment position is fixed, and then the RIS deployment position is updated, and this is iterated until convergence.

[0081] It should be noted that for agent 1, at time t, in the environment state Next action , the action of agent1 includes not only the moving position action of the RIS next to the main base station, but also the phase action of the RIS, which can be expressed as: .

[0082] To ensure the stability of the system, the position action will not change once optimized, while the phase action needs to adapt to the channel state. If the position deployment action update frequency is the same as or even faster than the phase action update frequency, it will greatly affect the agent's judgment of the phase, and the two need to be updated at different time scales. This patent considers asynchronous time scale strategy updates. To ensure that the algorithm outputs the position and phase action optimization results converge, the RIS position deployment action update frequency is slow and the phase action update frequency is fast, that is, the RIS position deployment is fixed, and the RIS optimal phase is optimized. When the system reaches a relatively stable state, the RIS position deployment action is updated, and it is iterated until the algorithm converges. In this way, not only the optimal phase can be found, but also the optimal deployment can be found asynchronously.

[0083] Based on this, the embodiment of the present invention further proposes an asynchronous update strategy, which designs two actor networks in agent 1, outputting position action and phase action respectively, and the update frequencies of the two actors are inconsistent. Actor 1 outputting position action is updated on a large time scale, while actor 2 outputting phase action is updated on a small time scale, so as to achieve the effect of asynchronous update. The framework is as follows Figure 4 shown.

[0084] It should be noted that the above embodiments and their features can be arbitrarily combined without conflict, and the new technical solutions obtained by the combination still fall within the protection scope of the present invention.

[0085] Embodiment 3 Another aspect of the present invention is to provide a multi-RIS assisted wireless network coverage optimization device, please refer to Figure 5 , shown is a multi-RIS-assisted wireless network coverage optimization device in the third embodiment of the present invention, the multi-RIS-assisted wireless network coverage optimization device comprises: A model building module 11, used to build a channel model of the multi-RIS-assisted wireless communication network according to the system architecture of the multi-RIS-assisted wireless communication network; A function construction module 12, for constructing a target reward function based on a channel model of the multi-RIS-assisted wireless communication network; A problem construction module 13 is used to construct an optimization problem with the goal of maximizing the target reward function, the deployment position and phase angle of each RIS as optimization variables, and the deployment range and phase angle range of each RIS as constraints; The coverage optimization module 14 is used to adopt a preset deep deterministic policy gradient algorithm to iteratively solve the optimization problem to obtain the optimized deployment position and phase angle of each RIS, so as to optimize the network coverage of the multi-RIS-assisted wireless communication network.

[0086] The functions or operation steps implemented when the above modules and units are executed are generally the same as those in the above method embodiments, and will not be repeated here.

[0087] Embodiment 4 Another aspect of the present invention also provides a multi-RIS assisted wireless network coverage optimization device, see Figure 6 , shown is a multi-RIS-assisted wireless network coverage optimization device in the fourth embodiment of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, the multi-RIS-assisted wireless network coverage optimization method as described above is implemented.

[0088] Among them, the multi-RIS-assisted wireless network coverage optimization device can specifically be a cloud server or an enterprise internal server, etc., and the processor 10 in some embodiments can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, which is used to run the program code stored in the memory 20 or process data, such as executing access restriction programs.

[0089] The memory 20 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 20 may be an internal storage unit of the multi-RIS-assisted wireless network coverage optimization device, such as a hard disk of the multi-RIS-assisted wireless network coverage optimization device. In other embodiments, the memory 20 may also be an external storage device of the multi-RIS-assisted wireless network coverage optimization device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. equipped on the multi-RIS-assisted wireless network coverage optimization device. Further, the memory 20 may also include both an internal storage unit and an external storage device of the multi-RIS-assisted wireless network coverage optimization device. The memory 20 may be used not only to store application software and various types of data installed in the multi-RIS-assisted wireless network coverage optimization device, but also to temporarily store data that has been output or is to be output.

[0090] It should be pointed out that Figure 4 The structure shown does not constitute a limitation on the multi-RIS-assisted wireless network coverage optimization device. In other embodiments, the multi-RIS-assisted wireless network coverage optimization device may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0091] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the multi-RIS-assisted wireless network coverage optimization method as described above is implemented.

[0092] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0093] More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0094] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0095] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0096] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multi-RIS-assisted wireless network coverage optimization method, It is characterized in that Applied in a multi-RIS-assisted wireless communication network, the method comprises the following steps: Establishing a channel model of the multi-RIS-assisted wireless communication network according to the system architecture of the multi-RIS-assisted wireless communication network; constructing a target reward function based on a channel model of the multi-RIS-assisted wireless communication network; The optimization problem is constructed with the goal of maximizing the target reward function, the deployment position and phase angle of each RIS as optimization variables, and the deployment range and phase angle range of each RIS as constraints; A preset deep deterministic policy gradient algorithm is used to iteratively solve the optimization problem to obtain the optimized deployment position and phase angle of each RIS, so as to optimize the network coverage of the multi-RIS-assisted wireless communication network.

2. The multi-RIS-assisted wireless network coverage optimization method according to claim 1, It is characterized in that The channel model of the multi-RIS-assisted wireless communication network includes: Base station to grid point si Channel gain ha,si , Channel gain from base station to RIS Ha,r , and RIS to grid points si Channel gain ɡr,si , specifically defined as: in, si ∈{ s 1, s 2,……, sK },in K is the total number of grids within the service range of the base station, ε 1. ε 2 and ε 3 represent ha,si , Ha,r and ɡr,si The Rician factor of the channel link, L 1. L 2 and L 3 represent ha,si , Ha,r and ɡ r,si The path loss is Respectively ha,si , Ha,r and ɡr,si The LoS component, Respectively ha,si , Ha,r and ɡr,si The NLoS component of 3. The multi-RIS-assisted wireless network coverage optimization method according to claim 1, It is characterized in that The target reward function is: in, Rr ( t ) represents r The target reward function of a RIS is M Represents the number of RIS.

4. The multi-RIS-assisted wireless network coverage optimization method according to claim 3, It is characterized in that The multi-RIS-assisted wireless communication network includes a RIS for primary base station assistance and a RIS for interference base station assistance; Among them, the objective reward function of RIS for primary base station assistance is: R ( t )= R ( t ) RSRP + R ( t ) SINR Among them, the objective reward function of the RIS used to interfere with the base station assisted RIS is: R ( t ) = R ( t ) SINR in, in, In the formula, R ( t ) RSRP represents the target reward function for the reference signal received power, R ( t ) SINR The target reward function represents the signal-to-interference-plus-noise ratio, Representing grid points si The target reward function for the reference signal received power at , Representing grid points si The target reward function for the signal-to-interference-plus-noise ratio at , RSRP Representing grid points si The reference signal received power at SINR Representing grid points si The signal to interference plus noise ratio at THRSRP and THSINR is the corresponding threshold.

5. The multi-RIS-assisted wireless network coverage optimization method according to claim 4, It is characterized in that in: In the formula, Represents the main base station to the grid point si The channel gain matrix at Represents the interfering base station to the grid point si The channel gain matrix at Represents the RIS to the grid point assisted by the main base station si The channel gain matrix, Represents the RIS to the grid point assisted by the interfering base station si The channel gain matrix, represents the channel gain matrix from the primary base station to the primary base station-assisted RIS, represents the channel gain matrix from the interfering base station to the interfering base station assisted RIS, and represents the diagonal matrix of the primary base station and the interfering base station, x 1 and x 2 represent the amount of user data to be transmitted by the primary base station and the interfering base station respectively, v 1 represents the dynamic noise caused by the RIS assisted by the main base station, n Represents environmental noise.

6. The multi-RIS-assisted wireless network coverage optimization method according to claim 1, It is characterized in that The steps of using a preset deep deterministic policy gradient algorithm to iteratively solve the optimization problem include: Every preset time t , control each RIS in the current state Select actions under the environment and the current policy network and execute; Get the actions being performed by each RIS Rewards returned from the later environment And observe the next state , to obtain multiple sets of training samples , and multiple sets of training samples Stored in the experience replay buffer; A batch of training samples is randomly extracted from the experience replay buffer, and the Q value of each RIS in the current policy network is calculated. The parameters of the policy network are iteratively updated by minimizing the mean square error until the network converges, and the final network parameters are output to obtain the optimized deployment position and phase angle of each RIS.

7. The multi-RIS-assisted wireless network coverage optimization method according to claim 6, It is characterized in that For the RIS for primary base station assistance, the steps of updating the parameters of the policy network include: Adopting the preset policy asynchronous update rules to asynchronously update the deployment position and phase angle of the RIS of the policy network; Among them, the strategy asynchronous update rule is to make the deployment position update frequency of RIS slower than the phase angle update frequency. Specifically, the phase angle is optimally solved when the deployment position of RIS is fixed, and then the deployment position of RIS is updated, and it is iterated until convergence.

8. A multi-RIS-assisted wireless network coverage optimization device, It is characterized in that The device comprises: A model building module, used to build a channel model of the multi-RIS-assisted wireless communication network according to the system architecture of the multi-RIS-assisted wireless communication network; A function construction module, for constructing a target reward function based on a channel model of the multi-RIS-assisted wireless communication network; The problem construction module is used to construct an optimization problem with the goal of maximizing the target reward function, the deployment position and phase angle of each RIS as optimization variables, and the deployment range and phase angle range of each RIS as constraints; The coverage optimization module is used to iteratively solve the optimization problem using a preset deep deterministic policy gradient algorithm to obtain the optimized deployment position and phase angle of each RIS, so as to optimize the network coverage of the multi-RIS-assisted wireless communication network.

9. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and when the program is executed by a processor, the multi-RIS-assisted wireless network coverage optimization method as claimed in any one of claims 1 to 7 is implemented.

10. A multi-RIS-assisted wireless network coverage optimization device, It is characterized in that The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the multi-RIS-assisted wireless network coverage optimization method as claimed in any one of claims 1 to 7 is implemented.

Images