User-centric Cell-free Massive MIMO Power Allocation Method Based on Offline Reinforcement Learning

Abstract

The present invention discloses a user-centric non-cellular large-scale MIMO power allocation method based on offline reinforcement learning, including: constructing a user-centric MIMO system, establishing a service relationship between a wireless access point and some users; taking the power control coefficient of the downlink as an optimization parameter to raise a question and construct a Markov decision process model; establishing a DuelingDDQN network and performing online training, storing state transition data generated by the interaction between the environment and the network during the online training process; taking out 20% of the online data set, and introducing a regular term in the loss function to perform offline training on the network. The "user-centric" power allocation strategy selects some users to access the wireless access point; the offline algorithm proposed by the present invention reduces the training cost, and only needs to deploy 20% of the online training data set for training, so as to achieve offline and real-time adjustment of the power control coefficient in real scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of non-cellular large-scale MIMO power allocation, and in particular relates to a user-centric non-cellular large-scale MIMO power allocation method based on offline reinforcement learning. Background Art

[0002] Wireless communication services permeate all walks of life in society, from ordinary answering calls and sending text messages to some emerging fields such as unmanned driving and smart medical care. All services, big and small, rely on the deployment of wireless networks. In order to ensure the quality of service, wireless communication services need to cover a large geographical area. Traditional wireless communication services use cellular network topology to deploy base stations, each of which serves a group of user devices. This cellular network topology has been used for decades. By reducing the size of cells and applying advanced signal processing solutions, user interference in this scenario can be reduced. In recent years, a new network topology has emerged in the field of wireless network services, called "cell-free massive MIMO system". In the cell-free massive MIMO system, the division of cells is cancelled, and the number of base stations is far greater than the number of users. The idea of ​​cell-free massive MIMO is to deploy a large number of distributed single-antenna access points (APs) and connect them through a central processing unit (CPU). The CPU runs the system in a cell-free boundary network MIMO mode and uses a cooperative transmission and reception method to serve users together. Compared with traditional cellular massive MIMO networks, non-cellular MIMO solutions have powerful macro-diversity capabilities, the ability to suppress multi-user interference, and the ability to provide users with services of similar quality. In recent years, they have received widespread attention and deployment.

[0003] However, there are also some problems with the non-cellular MIMO system. Since all APs and UEs in the system are fully connected, the large amount of power consumption in the fronthaul link has a significant impact on the energy efficiency of the non-cellular MIMO system, especially in the multi-antenna scenario. As the number of antennas increases, the power consumption of the fronthaul link also increases further, resulting in a decrease in the energy efficiency of the system. In addition, in order to further improve the user transmission rate and thus improve the user experience, the non-cellular MIMO system adopts a power allocation strategy to design the power control coefficient. Traditional power control methods require the establishment of an accurate model for the problem and then iterate to obtain the optimal solution. The time complexity of such algorithms is often very high and requires a lot of computing resources. With the development of modern computing resources, many algorithms based on deep neural networks have emerged. Existing power allocation strategies based on deep reinforcement learning all use online training strategies. This algorithm requires real-time interaction with the environment while training the network to obtain more data sets. However, in actual application scenarios, the interaction between the agent and the environment can usually only occur within a certain time interval, and real-time interaction is unrealistic. Therefore, these algorithms are usually not put into practical application. Summary of the invention

[0004] The present invention aims to provide a user-centric non-cellular massive MIMO power allocation method based on offline reinforcement learning, and an offline reinforcement learning power allocation method based on Dueling DDQN network for the downlink data transmission phase in the user-centric non-cellular massive MIMO scenario. After modeling the user-centric non-cellular massive MIMO environment, establishing the MDP model, online training of the Dueling DDQN network, and offline training of the Dueling DDQN network, the power control coefficient of the user-centric non-cellular massive MIMO is finally obtained to solve the technical problems mentioned in the background technology.

[0005] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0006] A user-centric cell-free massive MIMO power allocation method based on offline reinforcement learning comprises the following steps:

[0007] Step S1, modeling a user-centric non-cellular massive MIMO system, determining the service relationship between a wireless access point AP and a user equipment UE based on uplink channel estimation, optimizing the power control coefficient in the downlink data transmission phase, and maximizing the sum of downlink rates to propose an optimization problem;

[0008] Step S2: according to the optimization problem described in step S1, the optimization process of the power control coefficient in the downlink data transmission phase is modeled as a Markov decision process, and the state transition, action space, strategy and reward of the Markov decision process are determined;

[0009] Step S3, proposing a power allocation algorithm model based on deep reinforcement learning, the model includes a large-scale MIMO system environment module and an intelligent agent module; the large-scale MIMO system environment module is used to simulate the channel model of the user-centric non-cellular large-scale MIMO system and the downlink data transmission model, the intelligent agent module is used to perceive the current system state, estimate the Q value of the power allocation strategy and select the optimal power allocation coefficient; the core of the intelligent agent module is a deep neural network, and the training method of the deep neural network includes early online training and offline training in the application period;

[0010] Step S4, online training of the deep neural network; in the online training stage, before training the deep neural network based on the parameters in the data set, it is necessary to first collect the state transition parameters to update the data set; after initializing the large-scale MIMO system, first input the state of the system into the deep neural network, then select the power control coefficient for the current AP based on the Q value output by the deep neural network, implement the power control strategy in the environment, thereby changing the environmental state and obtaining a reward, and store the state transition information of this time; then randomly extract a batch of data from the data set, use the deep neural network to calculate the cumulative reward value and the expected value respectively, and update the deep neural network parameters with the goal of minimizing the mean square error between the reward value and the expected value;

[0011] Step S5, offline training Dueling DDQN network based on the state transition data set collected in step S4; taking out the first 20% of the state transition data set in step S4 as the offline training data set, taking out a batch of data from the offline data set each time and inputting it into the deep neural network, using the deep neural network to calculate the cumulative reward value and the expected value respectively, and updating the deep neural network parameters with the goal of minimizing the mean square error between the reward value and the expected value, so that the intelligent agent module finally selects the optimal power control coefficient.

[0012] Furthermore, in step S1, constructing a user-centric massive MIMO system specifically includes:

[0013] Step S101: First, a distribution area of ​​a scene is set, in which M APs and K UEs are randomly distributed, each AP is to serve N UEs, and then a large-scale fading model and a small-scale fading model of a channel between the AP and the UE are established;

[0014] Step S102: Modeling the uplink training phase, specifically including:

[0015] First, an orthogonal pilot sequence is allocated to the UE, and then the UE is instructed to forward the pilot sequence to each AP. After receiving the data, the AP estimates the channel coefficient between the AP and the UE based on the minimum mean square error criterion.

[0016] Step S103: Associating each AP with the UE that needs to be served, specifically including:

[0017] For each AP, the channel estimation coefficients between the AP and all UEs are arranged in descending order, and a UE with the highest channel coefficient is selected for each AP to establish a service relationship, and the established service relationship information is forwarded to the CPU;

[0018] Step S104, modeling the downlink data transmission phase, specifically includes:

[0019] The AP regards the channel estimation obtained in step S102 as the real channel coefficient, performs conjugate beamforming on the data to be transmitted, and then sends the precoded data to the UE that has established a connection relationship with the current AP with power.

[0020] Furthermore, in step S1, the optimization problem in step S1 is constructed based on the user signal-to-noise ratio, transmission rate and power limitation conditions in the downlink data transmission phase.

[0021] Furthermore, the user signal-to-noise ratio expression in the downlink data transmission phase is:

[0022]

[0023] In the formula, SINR k , k=1,…,K represents the signal-to-noise ratio of the kth user, β mk represents the large-scale fading of the channel between the mth AP and the kth UE; represents the normalized signal-to-noise ratio of the pilot symbol, represents the pilot sequence of the kth UE, n mk ,m=1,…,M,k=1,…K represents the power control coefficient between the mth AP and the kth UE, P(k),k=1,…K represents the set of APs serving the kth user; in the formula, in represents the minimum mean square error estimation of the channel between the mth AP and the kth UE, τ cf represents the number of uplink training samples in a coherent interval, c mk The expression is:

[0024]

[0025] Furthermore, the transmission rate of the downlink data transmission phase is expressed as:

[0026]

[0027] In the formula, represents the transmission rate of the kth UE, SINR k ,k=1,…,K represents the downlink signal-to-noise ratio of k UEs.

[0028] Furthermore, the expression of the power allocation optimization problem is:

[0029]

[0030] η mk ≥0,k=1,...,K,m=1,...,M;

[0031] Wherein, T(m), m=1, ..., M represents the index set of APs that establish a connection relationship with the mth AP, and there are N indexes in the set, indicating that each AP serves N UEs.

[0032] Furthermore, the step S2 models the optimization process of the power control coefficient in the downlink data transmission phase as a Markov decision process, specifically including:

[0033] Step S201: Modeling the optimization step of the power allocation coefficient in the system as a sequential decision-making process, wherein the elements of the process include states, actions, transfer strategies, and rewards; in the process, each step selects a power allocation coefficient for an AP in a user-centric massive MIMO system;

[0034] Step S202: Set the system state. The system state describes the signal-to-noise ratio of the user under the current power allocation strategy and specifies the AP optimization power control coefficient at the current moment. Set the current system state to specify the mth AP to update the power control coefficient, and the parameter η will be updated. mk ,k∈T(m);

[0035] Step S203: setting an action space, where the action space is a finite set, and the numbers in the set describe all optional values ​​of the power control coefficient;

[0036] Step S204, setting a state transition probability, where the state transition probability describes the probability that the environment changes to a new state after the power allocation strategy is implemented for the user-centric massive MIMO system, and its value is [0,1];

[0037] Step S205: setting a reward, where the reward describes the gain of the sum of transmission rates of K users after the power allocation strategy is implemented in the user-centric massive MIMO system.

[0038] Furthermore, the system state in step S202 is represented by s t =[SINR,c]∈S, where SINR is the user signal-to-noise ratio, which is a K-dimensional vector; the specific expression is:

[0039] SINR=[SINR1,...,SINR k ,...,SINR K ],

[0040] c is a one-hot encoding used to indicate the AP index value, and its specific expression is: e m is an M-dimensional vector with the mth dimension being 1 and the remaining dimensions being 0, indicating that the power control coefficient is currently to be updated for the mth AP. At the current moment, the intelligent experience updates the parameter η for the user-centric massive MIMO environment mk ,k∈T(m), That is, for UEs that have established a service relationship with the mth AP, the power control coefficients between these UEs and the current AP are updated; and for UEs that have not established a service relationship with the mth AP, the power control coefficients between them and the mth AP are set to 0;

[0041] Step S203: The action space is a t =(η m1 ,η m2 ,…,η mK ),in, It describes that the power coefficient of the UE that has no service relationship with the AP can only be 0; η mk ∈{0.1,0.4,0.7,1.0}, m=1,...,M,k∈T(m) describes the optional value of the power control coefficient of the UE that establishes a service relationship with the AP.

[0042] Furthermore, the online training process of the Dueling DDQN network in step S4 specifically includes:

[0043] Step S401, initializing the user-centric massive MIMO system environment module, that is, determining the distribution of APs and UEs and the channel model; initializing the agent module, that is, initializing the parameters and buffer area of ​​the Dueling DDQN network;

[0044] Step S402, collecting state transfer data; firstly, inputting the system state into the intelligent agent module, the intelligent agent module estimates the Q value of the current state, then selecting the power allocation coefficient based on the Q value, and transmitting the selected power control coefficient to the large-scale MIMO system environment module for implementation, thereby changing the environment state and obtaining the user signal-to-noise ratio gain as a reward, and finally saving the parameters of this state transfer into the buffer area;

[0045] Step S403, training the network; randomly extracting a batch of state transfer parameters from the buffer area, first taking the system state before the transfer as the input of the agent module, so that the agent perceives the state and estimates the cumulative reward value; then taking the state after the state transfer as the input of the agent module, so that the agent perceives the state and obtains the expected cumulative reward value in combination with the reward value information in the state transfer;

[0046] Step S404: With the goal of minimizing the mean square error between the cumulative benefit and the expected value, the network parameters of the Dueling DDQN network are updated using the back propagation algorithm; the agent-environment interaction operations of steps S402 to S403 are continuously repeated to continuously update the network parameters and the data set.

[0047] Furthermore, the offline training process in step S5 specifically includes:

[0048] Furthermore, the offline training process in step S5 specifically includes:

[0049] Step S501, initialize the user-centric massive MIMO system environment module, that is, determine the distribution of APs and UEs and the channel model; initialize the agent module, first randomly initialize the parameters of the Dueling DDQN network, and then take out the first 20% of the state transition parameter data set collected in step S4 as the offline training data set;

[0050] Step S502: randomly extract a batch of state transfer parameters from the offline training data set, first use the system state before the transfer as the input of the agent module, so that the agent perceives the state and estimates the cumulative reward value; then use the state after the state transfer as the input of the agent module, so that the agent perceives the state and combines the reward value information in the state transfer to obtain the expected cumulative reward value; with the goal of minimizing the mean square error between the calculated cumulative benefit and the expected value, use the back propagation algorithm to update the network parameters of the Dueling DDQN network;

[0051] Step S503: Repeat step S502 continuously, and use the offline data set to update the parameters of the Dueling DDQN network until the user's signal-to-noise ratio gain converges to a certain value, and then stop network training.

[0052] The user-centric non-cellular large-scale MIMO power allocation method based on offline reinforcement learning of the present invention has the following advantages:

[0053] 1. Compared with the general non-cellular large-scale MIMO system model, the user-centric large-scale MIMO model of the present invention reduces system power consumption and improves system energy efficiency;

[0054] 2. Compared with the traditional optimization-based power allocation algorithm, the algorithm based on reinforcement learning in the present invention reduces the time complexity and time cost of calculation;

[0055] 3. Compared with the online reinforcement learning algorithm, the algorithm based on offline reinforcement learning in the present invention reduces the data set size to 20% of the online training, and can perform real-time power allocation for user-centric non-cellular large-scale MIMO systems in actual application scenarios where the environment and the intelligent agent cannot interact in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flowchart of a user-centric non-cellular massive MIMO power allocation method based on offline reinforcement learning provided in Embodiment 1 of the present invention;

[0057] Figure 2 This is a schematic diagram of the framework of the power allocation model of the offline reinforcement learning algorithm provided in Example 1 of the present invention;

[0058] Figure 3 A flowchart for establishing a user-centric non-cellular massive MIMO model provided in Embodiment 1 of the present invention;

[0059] Figure 4 A schematic diagram of a user-centric non-cellular massive MIMO system provided in Embodiment 1 of the present invention;

[0060] Figure 5 This is a flow chart of the online training Dueling DDQN network provided in Example 1 of the present invention;

[0061] Figure 6 This is a flow chart of the offline training Dueling DDQN network provided in Embodiment 1 of the present invention;

[0062] Figure 7 This is the Dueling DDQN network offline training curve provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0063] In order to better understand the purpose, structure and function of the present invention, the user-centric non-cellular large-scale MIMO power allocation method based on offline reinforcement learning of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0064] Example 1

[0065] See also Figure 1-Figure 7 This embodiment provides a user-centric non-cellular large-scale MIMO power allocation method based on offline reinforcement learning, specifically as follows Figure 1 As shown, the method comprises the following steps:

[0066] Step S1: constructing a user-centric cellular-free massive MIMO system, specifically including:

[0067] Firstly, the distribution area of ​​the scenario is set, and the number of randomly distributed wireless access points (APs) and user equipment (UEs) as well as the number of UEs to be served by each AP are set. Then, the large-scale fading and small-scale fading models of the channel between AP and UE are established.

[0068] Then, an orthogonal pilot sequence is assigned to the UE, and the UE is instructed to forward the pilot sequence to each AP. After receiving the data, the AP estimates the channel coefficient between the AP and the characteristic UE based on the minimum mean square error criterion. For each AP, the channel estimation coefficients between the AP and all UEs are arranged in descending order, and N UEs with higher channel coefficients are selected for each AP to establish a service relationship, and the established service relationship information is forwarded to the CPU.

[0069] The AP will perform conjugate beamforming on the data to be transmitted based on channel estimation, and then send the precoded data to the UE that has established a connection with the current AP at a specific power.

[0070] The power control coefficient between AP and UE in the downlink data transmission phase in the user-centric cellular-free massive MIMO system is taken as the optimization object, and the sum of the UE rates in the downlink phase is maximized. The power allocation optimization problem is constructed based on the user signal-to-noise ratio, transmission rate and power limitation conditions in the downlink data transmission phase.

[0071] The optimization process of power allocation coefficient is modeled as a Markov decision process, and the state transition, action space, strategy and reward of the Markov decision process are determined.

[0072] Step S2: Model the optimization process of the power allocation coefficient as a Markov decision process. The MDP model can be described by a quaternion, which can be expressed as That is, the state space S and the action space State transition probability award The details are as follows:

[0073] 1. State space S, describing the state of the user-centric cell-free massive MIMO system. t =[SINR,c]∈S, where SINR is the user signal-to-noise ratio, which is a K-dimensional vector. The specific expression is:

[0074] SINR = [SINR1, ..., SINR k , …, SINR K ],

[0075] c is a one-hot encoding used to indicate the AP index value, and its specific expression is: e m is an M-dimensional vector with the mth dimension being 1 and the remaining dimensions being 0, indicating that the power control coefficient is currently to be updated for the mth AP. At the current moment, the intelligent experience updates the parameter η for the user-centric massive MIMO environment mk ,k∈T(m), That is, for UEs that have established service relationships with the mth AP, the power control coefficients between these UEs and the current AP are updated; and for UEs that have not established service relationships with the mth AP, the power control coefficients between them and the mth AP are set to 0.

[0076] 2. Action Space Describe the power control coefficients that an agent can implement for a user-centric, non-cellular, massive MIMO system. In this embodiment, in, It describes that the power coefficient of the UE that has no service relationship with the AP can only be 0; η mk ∈{0.1,0.4,0.7,1.0}, m=1,...,M,k∈T(m) describes the optional value of the power control coefficient of the UE that establishes a service relationship with the AP.

[0077] 3. Transition probability between states The value range is [0,1].

[0078] In this embodiment, it is assumed that the state s t =[SINR,c t ] by updating the power control coefficient (η) in a user-centric cell-free massive MIMO environment. m1 ,η m2 ,…,η mK ), so that the environment is transferred to state s t+1 =[SINR′,ct+1 ].

[0079] 4. Revenue Information In this embodiment, it is represented as That is, the gain of the sum of all user rates in a user-centric large-scale MIMO system before and after a state transfer.

[0080] Step S3, construct a power allocation algorithm model based on deep reinforcement learning, the model includes a user-centric large-scale MIMO system environment module and an intelligent agent module. The large-scale MIMO system environment module is used to simulate the channel model and downlink data transmission model of the user-centric non-cellular large-scale MIMO system, and the intelligent agent module is used to perceive the current system state, estimate the Q value of the power allocation strategy, and select the optimal power allocation coefficient; the core of the intelligent agent module is a deep neural network, and the training method of the deep neural network includes early online training and offline training during the application period.

[0081] Step S4, online training of Dueling DDQN network. In the online training phase, before training the network based on the parameters in the data set, it is necessary to first collect the state transition parameter update data set. After initializing the large-scale MIMO system, the state of the system is first input into the deep neural network, and then the power control coefficient is selected for the current AP based on the Q value output by the network, and the power control strategy is implemented in the environment, thereby changing the environmental state and obtaining rewards, and storing the state transition information this time. Then, a batch of data is randomly selected from the data set, and the network is used to calculate the cumulative reward value and the expected value respectively, and the network parameters are updated with the goal of minimizing the mean square error between the reward value and the expected value.

[0082] Step S5: Offline training of Dueling DDQN network based on the state transition data set collected in step S4. Offline training takes out the first 20% of the state transition data set described in step S4 as the offline training data set. Each time, a batch of data is taken out from the offline data set and input into the network. The network is used to calculate the cumulative reward value and the expected value respectively. The network parameters are updated with the goal of minimizing the mean square error between the reward value and the expected value, so that the intelligent agent module finally selects the optimal power control coefficient.

[0083] Specifically, in this embodiment, the specific structure of the power allocation algorithm model is as follows: Figure 2 As shown, more specifically, the power allocation model includes:

[0084] User-centric non-cellular massive MIMO environment module: simulates the large-scale fading and small-scale fading models of the channel, the uplink training model, the downlink data transmission model and the state transition of the MDP model, where the state transition mode of the MDP model includes rewards under different system states and power control coefficients.

[0085] Online training module: including cache area, Dueling DDQN network and action selection strategy. In the online training phase, before training the network based on the parameters in the data set, it is necessary to collect state transition parameters to update the data set. After initializing the large-scale MIMO system, the state of the system is first input into the deep neural network, and then the power control coefficient is selected for the current AP based on the Q value output by the network, and the power control strategy is implemented in the environment to change the environmental state and obtain rewards, and store the state transition information this time. Then, a batch of data is randomly selected from the data set, and the network is used to calculate the cumulative reward value and expected value respectively, and the network parameters are updated with the goal of minimizing the mean square error between the reward value and the expected value.

[0086] Offline training module: including offline training data set and Dueling DDQN network. Take out the first 20% of the online training buffer data set as the offline training data set. Take out a batch of data from the offline data set each time and input it into the network. Use the network to calculate the cumulative reward value and expected value respectively. Update the network parameters with the goal of minimizing the mean square error between the reward value and the expected value, and finally make the intelligent agent module select the best power control coefficient. The update of the offline training module network depends entirely on the training data sampled from the cache area, without the need for additional interaction with the environment.

[0087] Specifically, in this embodiment, a specific non-cellular massive MIMO system is provided, and the model establishment process is as follows: Figure 3 As shown, more specifically, the non-cellular massive MIMO system is established through the following steps:

[0088] Step S101: Consider an area of ​​1 km 2 In a square area, M APs and K UEs are randomly distributed in the area, and each AP serves N UEs. Figure 4 The figure shows the situation when M=8, K=6, and N=2. The AP and UE have only one antenna, and the APs are connected to the CPU through an ideal backhaul network. mk The channel coefficient describing the mth AP and the kth UE is defined by the following formula:

[0089]

[0090] In the formula, h mk, m=1,...,M,k=1,...,K represents small-scale fading, which obeys independent complex Gaussian distribution; β mk , m=1,...,M, k=1,...,K represents large-scale fading.

[0091] Step S102: Model the uplink training phase. First, an orthogonal pilot sequence is allocated to the UE, and then the UE is instructed to forward the pilot sequence to each AP. After receiving the data, the AP estimates the channel coefficient between the AP and the specific UE based on the minimum mean square error criterion. The channel estimation can be expressed as:

[0092]

[0093] In the formula, represents the minimum mean square error estimate of the channel between the mth AP and the kth UE, is the received signal y of the mth AP p,m The pilot signal of the kth UE The projection on. mk The expression is:

[0094]

[0095] Step S103: associate each AP with the UE it needs to serve. For each AP, arrange the channel estimation coefficients between the AP and all UEs in descending order, select N UEs with higher channel coefficients for each AP to establish service relationships, and forward the established service relationship information to the CPU. For example, for the mth AP, there are:

[0096]

[0097] Then the mth AP is user s m1 ,s m2 ,…,s mN service, that is, T(m) = {s m1 ,s m2 ,...,s mN}, while for other users s m,N+1 ,…,s mK , the mth AP does not transmit data to it, then

[0098] Step S104: In the downlink data transmission phase, the AP regards the channel estimation obtained in step S102 as the real channel coefficient, performs conjugate beamforming on the data to be transmitted, and then sends the precoded data to the UE connected to the current AP at a specific power. The data received by the kth UE can be expressed as:

[0099]

[0100] In the formula, r d,k represents the data received by the kth UE during the downlink data transmission phase, P(k) represents the set of APs serving the kth user, and q k , k=1,..., K represents the symbol to be sent to the kth UE, q k satisfy is an additive complex Gaussian noise with mean 0 and variance 1, that is, Power control coefficient η mk The following restrictions apply:

[0101]

[0102] As mentioned above, in the formula,

[0103] Step S105: The power allocation problem in the downlink data transmission phase of the user-centric non-cellular massive MIMO system is written as:

[0104]

[0105] η mk ≥0,k=1,...,K,m=1,...,M.

[0106] In the formula, represents the transmission rate of the kth UE, SINR k , k=1,...,K represents the downlink signal-to-noise ratio of the kth UE, which can be specifically expressed as:

[0107]

[0108] Specifically, in this embodiment, Figure 5 The specific process of online training of Dueling DDQN network is shown. Including:

[0109] Step S401, initialize the user-centric massive MIMO system environment module, that is, determine the distribution of APs and UEs and the channel model; initialize the agent module, that is, initialize the parameters and cache area of ​​the Dueling DDQN network.

[0110] Step S402, collecting state transition data. First, the system state is input into the intelligent agent module, the intelligent agent module estimates the Q value of the current state, and then selects the power allocation coefficient based on the Q value, and transmits the selected power control coefficient to the large-scale MIMO system environment module for implementation, thereby changing the environment state and obtaining the user signal-to-noise ratio gain as a reward, and finally saving the parameters of this state transition to the cache area.

[0111] Step S403, training the network. Randomly extract a batch of state transition parameters from the buffer area, first use the system state before the transition as the input of the agent module, so that the agent perceives the state and estimates the cumulative reward value; then use the state after the state transition as the input of the agent module, so that the agent perceives the state and combines the reward value information in the state transition to derive the expected cumulative reward value.

[0112] Step S404: With the goal of minimizing the mean square error between the cumulative benefit and the expected value, the network parameters of the Dueling DDQN network are updated using the back propagation algorithm. The agent-environment interaction operations of S402 to S403 are continuously repeated to continuously update the network parameters and the data set.

[0113] Specifically, in this embodiment, Figure 6 The specific process of offline training of Dueling DDQN network is shown. Including:

[0114] Step S501, initialize the user-centric large-scale MIMO system environment module, that is, determine the distribution of APs and UEs and the channel model; initialize the intelligent agent module, first randomly initialize the parameters of the Dueling DDQN network, and then take out the first 20% of the state transition parameter data set collected in step S4 as the offline training data set.

[0115] Step S502: Randomly extract a batch of state transfer parameters from the offline training data set, first use the system state before the transfer as the input of the agent module, so that the agent perceives the state and estimates the cumulative reward value; then use the state after the state transfer as the input of the agent module, so that the agent perceives the state and combines the reward value information in the state transfer to derive the expected cumulative reward value. With the goal of minimizing the mean square error between the calculated cumulative benefit and the expected value, the network parameters of the Dueling DDQN network are updated using the back propagation algorithm.

[0116] Step S503: Repeat step S502 continuously and use the offline data set to update the parameters of the Dueling DDQN network until the number of training steps reaches 10,000 rounds.

[0117] Specifically, in this embodiment, the cumulative benefit curve of the Dueling DDQN network trained using the offline reinforcement learning algorithm is as follows: Figure 7 shown. Figure 7 It represents the offline training curve of the network when M=10, K=6, N=4. Figure 7The horizontal axis represents the number of training rounds, and the vertical axis represents the normalized benefit value. When the training reaches 200 rounds, the reward value begins to stabilize, and until the 400th round, the reward value basically converges to 0.71. This shows that the user-centric large-scale MIMO power allocation algorithm based on offline reinforcement learning proposed in this embodiment can also obtain a good convergence effect when trained on only 20% of the online training buffer, and can design a suitable power allocation coefficient, which is helpful to improve the energy efficiency of the user-centric non-cellular large-scale MIMO system.

[0118] In summary, the present invention realizes a power allocation method in a user-centric cellular-free large-scale MIMO system based on offline reinforcement learning. By determining the system state, action space, transition probability and reward value in the optimization problem, the optimization problem process of the power allocation is modeled as an MDP model, and then an offline learning algorithm consisting of a user-centric cellular-free large-scale MIMO environment module, an online learning module, and an offline learning module is constructed. The parameters of the deep neural network are continuously optimized using the back propagation algorithm to obtain the power control coefficient that maximizes the sum of the user rates in the system. The present invention adopts a user-centric cellular-free large-scale MIMO system model, which can improve the energy efficiency of the system while ensuring the quality of service; the present invention proposes an algorithm based on offline reinforcement learning, uses an offline data set to train an intelligent agent, and obtains an allocation scheme for the power allocation coefficient. The algorithm only needs one online training, and can be deployed in actual scenarios to use offline training algorithms to achieve dynamic regulation of the power allocation coefficient.

[0119] The matters not described in detail in the present invention are all known technologies to those skilled in the art.

[0120] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A user-centric non-cellular massive MIMO power allocation method based on offline reinforcement learning, characterized in that: The following steps are involved: Step S1, modeling a user-centric non-cellular massive MIMO system, determining the service relationship between a wireless access point AP and a user equipment UE based on uplink channel estimation, optimizing the power control coefficient in the downlink data transmission phase, and maximizing the sum of downlink rates to propose an optimization problem; Step S2: according to the optimization problem described in step S1, the optimization process of the power control coefficient in the downlink data transmission phase is modeled as a Markov decision process, and the state transition, action space, strategy and reward of the Markov decision process are determined; Step S3, proposing a power allocation algorithm model based on deep reinforcement learning, the model includes a large-scale MIMO system environment module and an intelligent agent module; the large-scale MIMO system environment module is used to simulate the channel model of the user-centric non-cellular large-scale MIMO system and the downlink data transmission model, the intelligent agent module is used to perceive the current system state, estimate the Q value of the power allocation strategy and select the optimal power allocation coefficient; the core of the intelligent agent module is a deep neural network, and the training method of the deep neural network includes early online training and offline training in the application period; Step S4, online training of the deep neural Dueling DDQN network; In the online training phase, before training the deep neural network based on the parameters in the data set, it is necessary to collect the state transition parameters to update the data set; After initializing the massive MIMO system, the system state is first input into the deep neural network, and then the power control coefficient is selected for the current AP based on the Q value output by the deep neural network, and the power control strategy is implemented in the environment, thereby changing the environmental state and obtaining a reward, and storing the state transition information of this time; then a batch of data is randomly selected from the data set, and the cumulative reward value and the expected value are calculated respectively by the deep neural network, and the deep neural network parameters are updated with the goal of minimizing the mean square error between the reward value and the expected value; Step S5, offline training the Dueling DDQN network based on the state transition data set collected in step S4; Taking out the first 20% of the state transition data set in step S4 as an offline training data set, taking out a batch of data from the offline data set each time and inputting it into the deep neural network, using the deep neural network to calculate the cumulative reward value and the expected value respectively, and updating the deep neural network parameters with the goal of minimizing the mean square error between the reward value and the expected value, so that the intelligent agent module finally selects the optimal power control coefficient; In step S1, the construction of a user-centric massive MIMO system specifically includes: Step S101: First, a distribution area of ​​a scene is set, in which M APs and K UEs are randomly distributed, each AP is to serve N UEs, and then a large-scale fading model and a small-scale fading model of a channel between the AP and the UE are established; Step S102: Modeling the uplink training phase, specifically including: First, an orthogonal pilot sequence is allocated to the UE, and then the UE is instructed to forward the pilot sequence to each AP. After receiving the data, the AP estimates the channel coefficient between the AP and the UE based on the minimum mean square error criterion. Step S103: Associating each AP with the UE that needs to be served, specifically including: For each AP, the channel estimation coefficients between the AP and all UEs are arranged in descending order, N UEs with the highest channel coefficients are selected for each AP to establish a service relationship, and the established service relationship information is forwarded to the CPU; Step S104, modeling the downlink data transmission phase, specifically includes: The AP regards the channel estimation obtained in step S102 as the real channel coefficient, performs conjugate beamforming on the data to be transmitted, and then sends the precoded data to the UE that has established a connection relationship with the current AP with power; The step S2 models the optimization process of the power control coefficient in the downlink data transmission phase as a Markov decision process, specifically including: Step S201: Modeling the optimization step of the power allocation coefficient in the system as a sequential decision-making process, wherein the elements of the process include states, actions, transfer strategies, and rewards; in the process, each step selects a power allocation coefficient for an AP in a user-centric massive MIMO system; Step S202: Set the system state. The system state describes the signal-to-noise ratio of the user under the current power allocation strategy and specifies the AP optimization power control coefficient at the current moment. Set the current system state to specify the mth AP to update the power control coefficient, and the parameter η will be updated. mk ,k∈T(m); Step S203: setting an action space, where the action space is a finite set, and the numbers in the set describe all optional values ​​of the power control coefficient; Step S204, setting a state transition probability, where the state transition probability describes the probability that the environment changes to a new state after the power allocation strategy is implemented for the user-centric massive MIMO system, and its value is [0,1]; Step S205: setting a reward, where the reward describes the gain of the sum of K user transmission rates after the power allocation strategy is implemented in the user-centric massive MIMO system; The system state in step S202 is represented by s t =[SINR,c]∈S, where SINR is the user signal-to-noise ratio, which is a K-dimensional vector; the specific expression is: SINR=[SINR1,...,SINR k ,...,SINR K ], c is a one-hot encoding used to indicate the AP index value, and its specific expression is: e m is an M-dimensional vector with the mth dimension being 1 and the remaining dimensions being 0, indicating that the power control coefficient is currently to be updated for the mth AP. At the current moment, the intelligent experience updates the parameter η for the user-centric massive MIMO environment mk ,k∈T(m), That is, for UEs that have established a service relationship with the mth AP, the power control coefficients between these UEs and the current AP are updated; and for UEs that have not established a service relationship with the mth AP, the power control coefficients between them and the mth AP are set to 0; Step S203: The action space is a t =(η m1 ,η m2 ,…,η mK ),in, It describes that the power coefficient of the UE that has no service relationship with the AP can only be 0; η mk ∈{0.1,0.4,0.7,1.0}, m=1,...,M,k∈T(m) describes the optional value of the power control coefficient of the UE that establishes a service relationship with the AP; The online training process of the deep neural Dueling DDQN network in step S4 specifically includes: Step S401, initializing the user-centric massive MIMO system environment module, that is, determining the distribution of APs and UEs and the channel model; initializing the agent module, that is, initializing the parameters and buffer area of ​​the Dueling DDQN network; Step S402, collecting state transfer data; firstly, inputting the system state into the intelligent agent module, the intelligent agent module estimates the Q value of the current state, then selecting the power allocation coefficient based on the Q value, and transmitting the selected power control coefficient to the large-scale MIMO system environment module for implementation, thereby changing the environment state and obtaining the user signal-to-noise ratio gain as a reward, and finally saving the parameters of this state transfer into the buffer area; Step S403, training the network; randomly extracting a batch of state transfer parameters from the buffer area, first taking the system state before the transfer as the input of the agent module, so that the agent perceives the state and estimates the cumulative reward value; then taking the state after the state transfer as the input of the agent module, so that the agent perceives the state and obtains the expected cumulative reward value in combination with the reward value information in the state transfer; Step S404: With the goal of minimizing the mean square error between the cumulative benefit and the expected value, the network parameters of the Dueling DDQN network are updated using the back propagation algorithm; the agent-environment interaction operations of steps S402 to S403 are continuously repeated to continuously update the network parameters and the data set; The offline training process in step S5 specifically includes: Step S501, initialize the user-centric massive MIMO system environment module, that is, determine the distribution of APs and UEs and the channel model; initialize the agent module, first randomly initialize the parameters of the Dueling DDQN network, and then take out the first 20% of the state transition parameter data set collected in step S4 as the offline training data set; Step S502: randomly extract a batch of state transfer parameters from the offline training data set, first use the system state before the transfer as the input of the agent module, so that the agent perceives the state and estimates the cumulative reward value; then use the state after the state transfer as the input of the agent module, so that the agent perceives the state and combines the reward value information in the state transfer to obtain the expected cumulative reward value; with the goal of minimizing the mean square error between the calculated cumulative benefit and the expected value, use the back propagation algorithm to update the network parameters of the Dueling DDQN network; Step S503: Repeat step S502 continuously, and use the offline data set to update the parameters of the Dueling DDQN network until the user's signal-to-noise ratio gain converges to a certain value, and then stop network training.

2. The user-centric non-cellular massive MIMO power allocation method based on offline reinforcement learning according to claim 1, characterized in that: In the step S1, the optimization problem in step S1 is constructed based on the user signal-to-noise ratio, transmission rate and power limitation conditions in the downlink data transmission phase.

3. The user-centric non-cellular massive MIMO power allocation method based on offline reinforcement learning according to claim 2, characterized in that: The user signal-to-noise ratio expression in the downlink data transmission stage is: In the formula, SINR k , k=1,...,K represents the signal-to-noise ratio of the kth user, β mk represents the large-scale fading of the channel between the mth AP and the kth UE; represents the normalized signal-to-noise ratio of the pilot symbol, represents the pilot sequence of the kth UE, n mk ,m=1,...,M,k=1,...K represents the power control coefficient between the mth AP and the kth UE, P(k),k=1,...K represents the set of APs serving the kth user; in the formula, in represents the minimum mean square error estimation of the channel between the mth AP and the kth UE, τ cf represents the number of uplink training samples in a coherent interval, c mk The expression is:

4. The user-centric non-cellular massive MIMO power allocation method based on offline reinforcement learning according to claim 3 is characterized in that: The expression of the transmission rate in the downlink data transmission phase is: In the formula, represents the transmission rate of the kth UE, SINR k ,k=1,…,K represents the downlink signal-to-noise ratio of k UEs.

5. The user-centric non-cellular massive MIMO power allocation method based on offline reinforcement learning according to claim 4, characterized in that: The expression of the power allocation optimization problem is: or mk ≥0,k=1,…,K,m=1,…,M; Wherein, T(m), m=1, ..., M represents the index set of APs that establish a connection relationship with the mth AP, and there are N indexes in the set, indicating that each AP serves N UEs.

Images