RIS auxiliary enhanced communication method based on deep evolution strategy

Abstract

The invention provides an RIS auxiliary enhanced communication method based on a deep evolution strategy, and belongs to the technical field of wireless communication. According to the method disclosed by the invention, joint optimization is carried out on the transmitting end precoding and the RIS reflection phase shift matrix in the RIS auxiliary communication system through the proposed deep evolutionary strategy algorithm. Based on the strong fitting characteristic of deep learning and the interaction characteristic of reinforcement learning, the thought of an evolutionary algorithm is adopted, iterative optimization is carried out by taking a population containing different feature agents as a unit, and new genetic information is brought to the iterative process through recombination and variation. Compared with a traditional optimization algorithm, the method has a simple and clear optimization framework and low calculation complexity, and meanwhile, compared with a deep reinforcement learning algorithm based on gradient descent, the method has faster convergence and better system performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communication, and in particular to a RIS-assisted enhanced communication method based on a deep evolution strategy, and specifically to a method for jointly optimizing a transmitter precoding matrix and a RIS-end reflection phase shift matrix using a deep evolution strategy (DEP). Background Art

[0002] RIS is a sub-wavelength-thick artificial electromagnetic functional surface, composed of multiple independent and reconfigurable passive reflective units. It is easy to design and manufacture, and its surface electromagnetic properties can be customized artificially. Different from traditional technical means that require passive adaptation to wireless propagation channels, the communication link assisted by RIS can change the characteristics of the wireless channel, making the wireless environment part of the design parameters of the communication system, thereby increasing the upper limit of the wireless channel capacity.

[0003] When optimizing the phase shift matrix of the RIS-assisted communication system, the traditional convex optimization algorithm is no longer applicable due to the non-convex constraints of the reflection unit and the coupling with the precoding matrix. The existing solutions mainly convert the non-convex optimization problem into a convex optimization problem or use an alternating optimization method to solve it, such as semidefinite relaxation (SDR), Riemann conjugate gradient (RCG), fractional programming (FP), etc. However, this optimization method usually requires high computational complexity, which is not feasible for endpoints with high timeliness requirements and insufficient hardware computing resources. In addition, in the case of multiple antennas at the transmitting end, it is necessary to jointly optimize the active transmit beamforming at the base station end and the passive reflective beamforming at the RIS end. The traditional optimization algorithm usually uses an alternating optimization method to iteratively optimize the two separately. The solution results depend on the selection of the initial value, and the computational complexity increases sharply with the complexity of the communication system, which is inefficient for large-scale systems.

[0004] With the development of computer science and artificial intelligence (AI) technology, various machine learning (ML) algorithms have been widely used in communication systems to solve complex nonlinear non-convex optimization problems. At present, the application of deep learning (DL) in RIS-assisted communication systems mainly includes resource management, secure communication, beamforming and channel estimation. DL-based methods can significantly reduce complexity and computing time by using offline prediction, but they need to obtain a large number of valid data sets before training, which brings certain hardware storage pressure. At the same time, high-quality labels in the data sets are usually difficult to obtain. Unlike DL, Deep Reinforcement Learning (DRL) is an online training algorithm and does not require labels to be obtained in advance. DRL mainly obtains empirical training data through the interaction between the agent and the environment, and continuously iterates and improves the agent during the interaction process. Gradient descent-based machine learning algorithms such as DL and DRL require minimization of the loss function and back propagation of the gradient. Given the complexity of the communication environment and the network, it may be very difficult to create an agent with correct hyperparameters. For example, in DRL, whether it is value function learning or policy function learning, the agent interacts with the environment, collects experience, and then uses backpropagation to improve the accuracy of the neural network (policy). Hyperparameters need to be carefully adjusted, including choosing the correct optimizer function, mini-batch size, and learning rate to ensure stable and successful training. At the same time, in order to use gradient descent and backpropagation in neural networks, the optimization model created must be differentiable, and some models may not be able to be trained by gradient descent due to lack of differentiability. Summary of the invention

[0005] In view of the high computational complexity of traditional optimization algorithms in RIS-assisted communication systems and the dependence of existing DRL optimization algorithms on gradient descent and hyperparameters, the present invention proposes a RIS-assisted enhanced communication method based on a deep evolution strategy. Based on the idea of ​​evolutionary learning, the present invention creates a population containing multiple intelligent agents with different characteristics in a single generation, and obtains the optimal intelligent agent strategy through the iterative method of the population through recombination and mutation. The optimization algorithm framework of the present invention is simple, the implementation complexity is low, and there is no need for gradient descent and back propagation. It is a new way of joint optimization of RIS-assisted communication systems.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A RIS-assisted enhanced communication method based on a deep evolution strategy includes the following processes:

[0008] Step 1: Obtain the channel status information of the reflection link at time t and Respectively represent the links between the base station and RIS and between RIS and user k; and initialize the generation G, population size P, random subspace proportion η and individual mutation rate μ; where 0≤η≤1;

[0009] Step 2: Generate an initial population of P agents, and set g = 1; the agents adopt a deep neural network structure, and each agent is represented by a parameter vector composed of different weights w and biases b in each neuron. The dimension of the parameter vector is D p The input of the deep neural network structure is the channel state information of the reflection link, and the output is the transmitter precoding matrix W (t) and the RIS reflection phase shift matrix Φ (t) ;

[0010] Step 3: The agent interacts with the environment to obtain the corresponding fitness score, i.e. the reward value r (t) , thus obtaining the matching population;

[0011] Step 4: According to the random subspace strategy and subspace ratio η, combined with the reward value, select parent agents Para1 and Para2 from the matching population;

[0012] Step 5: Continue to hybridize and recombine the parent agents until the number of hybrid offspring agents reaches the population size P;

[0013] Step 6: Traverse the individuals in the hybrid offspring population, perform mutations, and set g = g + 1;

[0014]

[0015] Among them, mut() represents the mutation process. is a hybrid offspring intelligence p; the mutation process is: randomly select μ*D from the parameter vector of a hybrid offspring intelligence p P The parameters are randomly mutated to obtain

[0016] Step 7: Return to step 3 until g = G;

[0017] Step 8: Output the transmitter precoding matrix W corresponding to the agent with the highest reward value at time t (t) and the RIS reflection phase shift matrix Φ (t) ;

[0018] Step 9: At time t, the base station uses the precoding matrix W (t) Send signal x (t) After processing, the processed transmission signal is incident on the RIS, and the RIS reflection phase shift matrix Φ (t)After regulation, it finally reaches the user, and the received signal of user k is expressed as is the transmitter precoding matrix W (t) The vector corresponding to user k in ,σ k is the noise at user k.

[0019] Furthermore, it is characterized in that the reward value r in step 3 t The calculation method is:

[0020]

[0021] In the formula, K is the total number of users, is the precoding matrix W at the transmitter at time t (t) The vector corresponding to user m in .

[0022] Furthermore, it is characterized in that the specific process of step 4 is:

[0023] According to the random subspace strategy, ηP agents are randomly selected from the matching population according to the subspace ratio η to form a subspace, and the agents are sorted according to their fitness scores. Then, the two agents with the highest fitness scores are selected as parent agents.

[0024] The present invention has the following advantages:

[0025] 1. The RIS-assisted enhanced communication method based on the deep evolution strategy proposed in the present invention jointly optimizes the precoding matrix of the transmitting end and the reflection phase shift matrix of the RIS end during the population iteration through the idea of ​​evolutionary learning. Compared with the traditional optimization algorithm, it is concise and clear, does not require complex formula derivation and alternating iterative optimization, and can simultaneously obtain the optimized precoding matrix and reflection phase shift matrix.

[0026] 2. Based on the strong fitting characteristics of deep learning and the interactive characteristics of reinforcement learning, the present invention performs iterative optimization through population recombination and mutation without gradient descent and back propagation. Compared with the existing deep reinforcement learning algorithm, it has stronger exploratory and faster convergence characteristics, and can achieve better system performance.

[0027] 3. The state space required by the present invention is only composed of channel state information. Compared with the DRL algorithm based on gradient descent, it does not require a large amount of interaction overhead. At the same time, in view of the strong exploratory nature of evolutionary learning, a random subspace strategy is proposed for parent selection, which effectively balances the exploratory nature and exploitability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a block diagram of the communication system of the present invention.

[0029] Figure 2 It is a flowchart of the optimization process of the method of the present invention.

[0030] Figure 3 It is a structural diagram of the deep neural network adopted in the present invention.

[0031] Figure 4 It is a single neuron model in the present invention.

[0032] Figure 5 It is the reorganization process of the parent agent in the present invention.

[0033] Figure 6 It is the mutation process of the progeny population in the present invention.

[0034] Figure 7 It is the curve of the algorithm performance of the present invention changing with the transmission power.

[0035] Figure 8 It is the convergence analysis curve of the algorithm of the present invention. DETAILED DESCRIPTION

[0036] The present invention is described in detail below with reference to the accompanying drawings.

[0037] A RIS-assisted enhanced communication method based on deep evolution strategy is proposed. This method is based on the strong fitting property of deep learning and the interactive characteristics of reinforcement learning. It adopts the idea of ​​evolutionary learning and obtains an intelligent agent strategy with optimal feature performance through iterative optimization of population recombination and mutation.

[0038] The communication system block diagram of the present invention is as follows: Figure 1 As shown, RIS assisted downlink communication is composed of N = 4 reflection units, the number of antennas at the base station transmitting end is M = 4, and there are K = 4 single-antenna users in total; the direct link is blocked, and communication is performed only by relying on the reflection link based on RIS.

[0039] Combination Figure 2 The iterative optimization flow chart of population recombination variation is shown, and the steps of the method of the present invention are described as follows:

[0040] Step 1: Figure 1 The system model shown in the figure reflects the link channel status information at time t. and Initialize the number of generations G = 50, the population size P = 500, the random subspace ratio η = 0.5, and the individual mutation rate μ = 0.001.

[0041] Step 2: Generate an initial population of P agents, and set g = 1. The agents of the present invention are all Figure 3 The deep neural network structure shown in Figure 1 (including two hidden layers with the same number of neurons) is unchanged, and each agent consists of Figure 4The parameter vector representation composed of different weights w and biases b in the neuron model shown in the figure can be obtained from the neural network structure to obtain the dimension D of the agent parameter vector p ; The input of the deep neural network structure is the channel state information of the reflection link, and the output is the transmitter precoding matrix W (t) and the RIS reflection phase shift matrix Φ (t) .

[0042] Step 3: The agent interacts with the environment to obtain the corresponding fitness score, i.e. the reward value r (t) , thus obtaining the matching population;

[0043]

[0044] In the formula, K is the total number of users, is the precoding matrix W at the transmitter at time t (t) The vector corresponding to user m in , is the transmitter precoding matrix W (t) The vector corresponding to user k in ,σ k is the noise at user k.

[0045] Step 4: According to the random subspace strategy and subspace ratio η, combined with the reward value, select parent agents Para1 and Para2 from the matching population; the specific process is:

[0046] According to the random subspace strategy, ηP agents are randomly selected from the matching population according to the subspace ratio η to form a subspace, and the agents are sorted according to their fitness scores. Then, the two agents with the highest fitness scores are selected as parent agents.

[0047] At this time, according to the size of η, there are two special cases: when η = 1, the subspace is the entire population space, and the random subspace strategy degenerates into a pure utilization strategy; when η = 2 / P, the random subspace strategy degenerates into a pure exploration strategy. Therefore, a reasonable choice of the size of η can better balance exploration and utilization in reinforcement learning.

[0048] Step 5: Figure 5 As shown, the parent agent is reorganized to obtain a hybrid child agent.

[0049]

[0050] Repeat the random recombination of parent agents until the number of hybrid offspring agents reaches the population size P. If the offspring population is only produced by parent recombination, all individuals in the population will soon become too similar to find the optimal strategy for the agent, so the mutation process is introduced.

[0051] Step 6: Figure 6 As shown, traverse the individuals of the hybrid offspring population, perform mutations, and set g = g + 1;

[0052]

[0053] Step 7: Return to step 3, the offspring population continues to interact with the environment to obtain a matching population until g=G.

[0054] Step 8: Output the transmitter precoding matrix W corresponding to the agent with the highest reward value at time t (t) and the RIS reflection phase shift matrix Φ (t) .

[0055] Step 9: Get the precoding matrix W (t) and the reflection phase shift matrix Φ (t) The signal transmission of the RIS-assisted communication system can be optimized. At time t, the transmitter (base station) uses the precoding matrix W (t) Send signal x (t) Processing, sending the signal to RIS, through the phase shift matrix Φ (t) The control finally reaches the receiving end (user). Then the received signal of user k can be expressed as

[0056] Principle description:

[0057] The basic idea of ​​the present invention is to use evolutionary algorithms based on deep reinforcement learning and optimize the agent strategy by using population iteration. First, according to the initial parameter settings, an initial population containing agents with different characteristics is generated, and the agents interact with the environment to obtain a matching population. The parent generation is selected for recombination and hybridization to generate a child population, and the child population continues to interact with the environment until the maximum number of generations is reached.

[0058] Under different transmission powers, the system performance change curves of different optimization schemes are as follows: Figure 7As shown. Three groups of baseline comparison schemes are set for simulation: 1. Deep deterministic policy gradient (DDPG) algorithm; 2. Weighted minimum mean square error (WMMSE) + Riemann conjugate gradient (RCG) algorithm; 3. WMMSE + RIS random phase shift. It can be seen that in the traditional optimization scheme, the WMMSE algorithm is used to optimize the beamforming matrix at the transmitting end. Compared with the random phase shift at the RIS end, the system performance is significantly improved after the RCG algorithm is used for joint optimization. Gradient-based DRL algorithms such as DDPG have certain advantages over traditional optimization algorithms. It can be seen that when the transmission power is 10dBm, the system sum rate is 2.139bps / Hz under the WMMSE+RCG algorithm, and the convergence value of the DDPG algorithm is 2.43bps / Hz. At the same time, due to the strong exploratory nature of the proposed DEP algorithm, it shows obvious superiority over the traditional optimization algorithm and the DDPG algorithm of single-agent iterative optimization, and its system sum rate convergence value is increased to 2.753bps / Hz.

[0059] Figure 8 The convergence of the proposed DEP algorithm is analyzed and compared with the gradient-based DDPG algorithm. It can be seen that the DDPG algorithm converges to a system and rate of about 2.43bps / Hz in about 30,000 steps. Its iterative process depends on the continuous interaction between a single agent and the environment, and improves the agent strategy through immediate rewards. Therefore, the convergence process is more volatile and the convergence speed is slow. Unlike DDPG, DEP relies on the simultaneous interaction of multiple agents and the environment in the population. The population size is defined as 500 here. In order to facilitate comparison with DDPG, the number of DEP generations G is set to 70. It can be seen that the DEP algorithm reaches convergence at the 5000th step, that is, the 10th generation, and the average reward of the population converges to about 2.75bps / Hz. Its convergence speed is much faster than that of the DDPG algorithm for single-agent iterative optimization.

[0060] In summary, the present invention jointly optimizes the transmitter precoding and RIS reflection phase shift matrix in the RIS-assisted communication system through the proposed RIS-assisted enhanced communication method based on deep evolution strategy. Based on the strong fitting characteristics of deep learning and the interactive characteristics of reinforcement learning, the idea of ​​evolutionary algorithm is adopted, and iterative optimization is performed in units of populations containing intelligent agents with different characteristics, and new genetic information is brought to the iterative process through recombination and mutation. Compared with traditional optimization algorithms, it has a concise and clear optimization framework and low computational complexity, and at the same time, it has faster convergence and better system performance than deep reinforcement learning algorithms based on gradient descent.

Claims

1. A RIS-assisted enhanced communication method based on deep evolution strategy, characterized in that: The process includes: Step 1: Obtain the channel status information of the reflection link at time t and Respectively represent the links between the base station and RIS and between RIS and user k; and initialize the generation G, population size P, random subspace proportion η and individual mutation rate μ; where 0≤η≤1; Step 2: Generate an initial population of P agents, and set g = 1; the agents adopt a deep neural network structure, and each agent is represented by a parameter vector composed of different weights w and biases b in each neuron. The dimension of the parameter vector is D p The input of the deep neural network structure is the channel state information of the reflection link, and the output is the transmitter precoding matrix W (t) and the RIS reflection phase shift matrix Φ (t) ; Step 3: The agent interacts with the environment to obtain the corresponding fitness score, i.e. the reward value r t , thus obtaining the matching population; Step 4: According to the random subspace strategy and subspace ratio η, combined with the reward value, select parent agents Para1 and Para2 from the matching population; Step 5: Continue to hybridize and recombine the parent agents until the number of hybrid offspring agents reaches the population size P; Step 6: Traverse the individuals in the hybrid offspring population, perform mutations, and set g = g + 1; Among them, mut() represents the mutation process. is a hybrid offspring intelligence p; the mutation process is: randomly select μ*D from the parameter vector of a hybrid offspring intelligence p P The parameters are randomly mutated to obtain Step 7: Return to step 3 until g = G; Step 8: Output the transmitter precoding matrix W corresponding to the agent with the highest reward value at time t (t) and the RIS reflection phase shift matrix Φ (t) ; Step 9: At time t, the base station uses the precoding matrix W (t) Send signal x (t) After processing, the processed transmission signal is incident to RIS, and the RIS reflection phase shift matrix Φ (t) After regulation, it finally reaches the user, and the received signal of user k is expressed as is the transmitter precoding matrix W (t) The vector corresponding to user k in , σ k is the noise at user k.

2. According to claim 1, a RIS-assisted enhanced communication method based on deep evolution strategy is characterized in that: The reward value r in step 3 t The calculation method is: In the formula, K is the total number of users, is the precoding matrix W at the transmitter at time t (t) The vector corresponding to user m in .

3. The RIS-assisted enhanced communication method based on deep evolution strategy according to claim 1 is characterized in that: The specific process of step 4 is as follows: According to the random subspace strategy, ηP agents are randomly selected from the matching population according to the subspace ratio η to form a subspace, and the agents are sorted according to their fitness scores. Then, the two agents with the highest fitness scores are selected as parent agents.

Images