Method for power allocation in massive MIMO system based on deep deterministic policy gradient

Abstract

The application discloses a large-scale MIMO system power distribution method based on deep deterministic policy gradient, and belongs to the field of intelligent wireless transmission. The method comprises the following steps: constructing a communication system model; building a deterministic policy gradient network structure; defining a training network state, an action, a reward function and a priority experience replay mechanism, and saving model weight parameters; loading training parameters and outputting an optimal power distribution scheme. Compared with a traditional power distribution algorithm, the method significantly improves the spectral efficiency of a communication system. Compared with an intelligent power distribution algorithm based on deep reinforcement learning, the method can significantly improve the adaptability of a complex wireless communication network, and can make a mobile communication network better cope with changing environments and demands, thereby providing more efficient and more reliable services.

Description

Technical Field

[0001] This invention belongs to the fields of deep reinforcement learning and wireless communication technology, and in particular to a multi-stream intelligent power allocation method for large-scale MIMO systems based on deep deterministic policy gradients. Background Technology

[0002] With the rapid development of mobile communication technology, next-generation wireless networks face higher demands for high speed, low latency, and high energy efficiency, leading to more complex, dense, and dynamic wireless networks. In this context, due to the limited nature of wireless network resources, efficient resource allocation is crucial for optimizing communication performance. Massive MIMO systems, with their spatial diversity advantages, can significantly increase the capacity of communication systems and are one of the key technologies for future communication systems. These systems typically employ large-scale antenna arrays at the transceiver end, resulting in a substantial increase in system power requirements. Therefore, the rational allocation of multi-stream power to limited power resources becomes a critical issue, directly impacting system performance and interference levels.

[0003] Power allocation technology is a strategy used in communication systems to dynamically adjust the allocation of power resources at the transmitter through optimization algorithms, thereby optimizing system performance such as spectral efficiency, energy efficiency, and quality of service. This technology can effectively improve spectrum utilization and power allocation efficiency while reducing the energy consumption and latency of communication systems, playing a crucial role in wireless networks, especially in massive MIMO communication systems.

[0004] Research on power allocation technology for massive MIMO systems has evolved from traditional optimization to intelligent optimization. The main method of traditional power allocation is to optimize one of the design criteria through mathematical programming, such as maximizing the sum-rate or minimizing interference, while imposing constraints on the remaining components. Currently, many model-oriented algorithms have been developed to manage interference to achieve sum-rate maximization, with existing research mainly focusing on suboptimal or heuristic algorithms. Although traditional power allocation algorithms have achieved significant success, many formulaic optimization problems remain difficult to solve, and it is challenging to balance and model different service demands mathematically. Furthermore, the wireless resource allocation optimization problem in most communication systems is non-convex, computationally complex, and difficult to execute in real-time; the performance gap between the solution and the optimal solution is often difficult to quantify.

[0005] The rise of deep learning technology in recent years has provided a new approach to power allocation problems in wireless communication. The most direct method for resource allocation using deep learning is to treat a given optimization problem as a black box and use various deep learning techniques to learn its input-output relationship. Leveraging the universal approximation property of deep neural networks, the mapping from the parameter inputs of a given optimization algorithm to the solution can be approximated. Compared to solving optimization problems using traditional methods, this significantly reduces computational complexity and produces real-time achievable solutions. However, since the learned labels must be provided by traditional optimization algorithms, the performance of the algorithm after convergence is limited by the performance of samples from traditional algorithms, resulting in lower practicality. Summary of the Invention

[0006] This invention proposes a power allocation method for large-scale MIMO systems based on deep reinforcement learning. Building upon the DDPG algorithm architecture of deep reinforcement learning, this method improves the dual-network architecture of the traditional DDPG algorithm by designing a priority-based empirical replay strategy, using adaptive Gaussian noise for action exploration, and optimizing the reward function, thereby enhancing the spectral efficiency of the power allocation algorithm.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A power allocation method for large-scale MIMO systems based on deep deterministic policy gradients includes the following steps:

[0009] S1: Construct a model of a large-scale MIMO communication system;

[0010] S2: Construct a power allocation algorithm based on DDPG and build a deterministic policy gradient network structure;

[0011] S3: Based on the deterministic policy gradient network structure of S2, the state space and action space of the training network are defined; the deterministic policy gradient network is trained using a reward function with constraints and a priority experience replay mechanism, and the model weight parameters are saved.

[0012] S4: Load the model weight parameters saved in step S3 and output the power allocation scheme.

[0013] Further, step S1 includes:

[0014] S11: If the transmitting end is equipped with M antennas and the receiving end is equipped with N antennas, using beamforming technology, and the number of transmitted data streams is K, then the number of data streams is:

[0015] ;

[0016] S12: Considering the same time-frequency resources, the channel from the transmitter to the receiver... Modeled as,

[0017]

[0018] in For a Rayleigh fading channel, g is the large-scale fading coefficient. According to the LTE standard, the large-scale fading coefficient g is modeled as...

[0019]

[0020] in Let be a log-normal random variable, following the... , It is the distance from the sender to the receiver. The standard deviation is the preset value.

[0021] S13: The transmitting end uses a beamforming matrix. The receiver uses a beamforming matrix. The power allocated to each corresponding stream signal is ,in For the transmit beamforming vector of the k-th data stream, For the receive beamforming vector of the k-th data stream, The power value allocated to the k-th data stream; ;

[0022] The SNR corresponding to the k-th data stream can be expressed as:

[0023]

[0024] in The power of additive white Gaussian noise, ,in For noise power density, Channel bandwidth;

[0025] The data rate of the k-th data stream can then be expressed as:

[0026] .

[0027] Furthermore, the specific method of step S2 includes:

[0028] A deterministic policy gradient network structure is constructed based on the DDPG algorithm, comprising a policy network and a value network. Both the policy network and the value network adopt a dual neural network structure containing an online network and a target network. The online policy network has the same structure as the target policy network, including an input layer, n1 sequential fully connected layers, and an output layer. The output layer of both the online policy network and the target policy network includes K neurons. The online value network has the same structure as the target value network, including an input layer, n2 sequential fully connected layers, and an output layer. The output layer of both the online value network and the target value network includes one neuron.

[0029] Furthermore, in step S3, based on the deterministic policy gradient network structure of S2, the state space and action space of the training network are defined; the specific method of using a reward function with constraints is as follows:

[0030] S31: Define the state space. In the communication system, the state information is set as the SNR of each data stream at the current time plus the power allocation coefficient of the previous time. The state matrix at time t is specifically represented as follows:

[0031]

[0032] in Let be the power distribution coefficient at time t-1. The power allocated to the k-th data stream at time t-1. When t=1, ; , Let SNR be the SNR of the k-th data stream at time t; , Let t be the channel from the transmitter to the receiver at time t;

[0033] S32: The power of a communication system is a non-negative continuous scalar, subject to the maximum power. Due to limitations, the DDPG online policy network receives the state matrix at time t. It directly outputs the corresponding deterministic action. :

[0034]

[0035] in This represents the output of the k-th neuron in the output layer of the online policy network. This refers to the k-th action output by the online policy network at time t.

[0036] S33: Environment Interaction: Adaptive Gaussian noise with exploration intensity decreasing with training steps is used.

[0037]

[0038] in, With a mean of 0, the covariance matrix is Multidimensional Gaussian noise, for An identity matrix of order 1. Let be the number of data streams actually transmitted at time t. , The standard deviation is adaptively adjusted during training. The standard deviation gradually decreases as the number of training steps increases. The formula for the decrease in step size is expressed as:

[0039]

[0040] in The initial standard deviation, This is the final standard deviation. The total number of training rounds makes the exploration efficiency higher in the early stage of training and gradually converges in the later stage.

[0041] The training process achieves the exploration of the action space by adding noise to the movements.

[0042]

[0043] in, These are set random parameters during training. The power values ​​allocated to the K data streams at time t+1 are as follows: This means controlling the sum of the power values ​​of the K data streams to the minimum transmit power. With maximum transmission power between;

[0044] The output of the online policy network after exploring the action space is denoted as:

[0045]

[0046] in, It is an online policy network. For parameters of the online policy network, This represents the state matrix received by the online policy network at time t. The output after the exploration behavior in the action space;

[0047] S34: Set the system reward value by adding a series of penalty terms to the average spectral efficiency. The current system reward value is specifically set as follows:

[0048]

[0049] in, , , , The percentage parameters for each reward component are all limited to the range [0,1]. , , Represents the time t at time t. The power value allocated to each actual data stream being transmitted; When t=1, .

[0050] Furthermore, in step S3, a deterministic policy gradient network is trained using a reward function with constraints and a priority experience replay mechanism. The specific method for saving the model weight parameters is as follows:

[0051] S35: The online policy network receives the state matrix at time t. Output the exploration behavior through the action space. Further obtain system reward value With the state matrix at the next time step ;

[0052] S36: Composition Sample The samples are stored in the experience replay pool. When the number of samples reaches the capacity threshold of the experience replay pool, the priority of the samples in the experience replay pool is adjusted by using time-series differential error. The priority experience replay mechanism is used to extract the samples of the preset batch with the highest priority according to the sample priority.

[0053] S37: Of the samples drawn ( The input to the online value network is , and the output of the online value network is the corresponding current action value. The online policy network is updated by maximizing the action value in the current state through gradient ascent.

[0054] S38: From the sample As input to the target policy network, refer to steps S32 and S33 to generate the exploration behavior through the action space corresponding to the state at the next time step. ,by( The target value network is input to the target value network, which generates the corresponding target action value. The parameters of the online value network are updated by minimizing the temporal difference error between the predicted action value and the target action value through gradient.

[0055] S39: Update the parameters of the target policy network and the target value network online through a soft update mechanism; for the preset batch samples in the current training round, record the maximum system reward value as the reward value of the current round;

[0056] S40: Repeat steps S35-S39 above, iteratively updating the policy network and value network until the total number of training rounds is reached. Then, save the online policy network model weight corresponding to the maximum reward value in each round.

[0057] Furthermore, step S37, which involves updating the online policy network by maximizing the action value at the current time state through gradient ascent, specifically includes:

[0058]

[0059]

[0060] in, Let the loss function be the online policy network. For the parameters of the online value network, The number of samples to be drawn in a pre-defined batch in the experience replay pool. To draw a sample set for a pre-defined batch, It is the state matrix at time t. To calculate the gradient of the online policy network parameters, To output the function of the online policy network Find the gradient. This indicates the value of the current action corresponding to the output of the online value network. Update according to the following formula:

[0061]

[0062] in, This is the learning rate of the online policy network; repeat step S37 until... convergence.

[0063] Furthermore, in step S38, the parameters of the online value network are updated by minimizing the temporal difference error between the predicted action value and the target action value using gradients, specifically as follows:

[0064]

[0065]

[0066] in, The loss function for the online value network, For the parameters of the target policy network, The parameters of the target value network; To calculate the gradient of the target value network parameters; It is a discount factor. The target policy network output represents the exploration behavior through the action space. The target value network outputs the corresponding target action value. Update as follows:

[0067]

[0068] in, It is the learning rate of the online value network; repeat step S38 until... convergence.

[0069] Furthermore, in step S39, the parameters of the target network are synchronously updated from the online network using a soft update mechanism, specifically as follows:

[0070]

[0071] in, It is a soft update coefficient, and ; and These are the parameters for the online policy network and the target policy network, respectively; and These are the parameters for the online value network and the target value network, respectively.

[0072] Furthermore, the specific method of step S4 is as follows:

[0073] The state matrix at the current moment Input the data into the saved online policy network model, and the online policy network model will directly output the corresponding deterministic actions. This is denoted as the output power distribution scheme. There are K elements consisting entirely of zeros.

[0074] The beneficial effects of this invention are as follows:

[0075] (1) This invention uses a deep reinforcement learning-based intelligent method for power allocation. Through the autonomous decision-making ability of the algorithm, it continuously optimizes in the interaction with the environment, significantly improving the adaptability of complex communication networks and enhancing spectrum performance.

[0076] (2) The present invention uses an improved deep deterministic strategy gradient algorithm for multi-stream power allocation, uses the dual network architecture of the DDPG algorithm for model training, and uses adaptive Gaussian noise with exploration intensity decaying with the number of training steps to explore the action space, so that the exploration efficiency of the training process is higher and the convergence effect is better.

[0077] (3) This invention adds a power penalty term to the reward function to avoid the situation where the power of individual data streams is zero, thereby further ensuring the fairness between data streams, accelerating the network convergence speed and improving training stability. Attached Figure Description

[0078] Figure 1 This is a flowchart of the power allocation method for large-scale MIMO systems based on deep deterministic policy gradients, as proposed in this invention.

[0079] Figure 2 This is a diagram of the DDPG algorithm framework of this invention.

[0080] Figure 3 This is a simulation result diagram comparing the power allocation method of this invention with other power allocation methods. Detailed Implementation

[0081] The present invention will now be described in detail with reference to the accompanying drawings.

[0082] This invention proposes a power allocation method for large-scale MIMO systems based on deep deterministic policy gradient (DDPG) algorithms. DRL-based power control algorithms determine power control schemes by observing the MIMO system environment, requiring no fixed dataset for training. This significantly improves the adaptability of complex wireless communication networks and enables mobile communication networks to better cope with constantly changing environments and demands. Combining deep reinforcement learning-based artificial intelligence technology, this invention addresses the performance limitations and high complexity of traditional power allocation algorithms, as well as the performance constraints and complex training samples of deep learning-based power allocation schemes imposed by traditional algorithms. Exploring intelligent multi-stream power allocation technology based on the deep deterministic policy gradient (DDPG) algorithm has significant theoretical and practical application value. Figure 1 As shown, it includes the following steps:

[0083] S1: Construct a model of a large-scale MIMO communication system;

[0084] S2: Construct a power allocation algorithm based on DDPG and build a deterministic policy gradient network structure;

[0085] S3: Based on the deterministic policy gradient network structure of S2, the state space and action space of the training network are defined; the deterministic policy gradient network is trained using a reward function with constraints and a priority experience replay mechanism, and the model weight parameters are saved.

[0086] S4: Load the model weight parameters saved in step S3 and output the power allocation scheme.

[0087] Further, step S1 includes:

[0088] The specific parameter configuration for the large-scale MIMO communication scenario model is as follows: the number of antennas at the transmitting end is M=32, the number of antennas at the receiving end is N=32, the maximum number of data streams K is 32, and the distance between the transmitting and receiving ends is d=10km; the log-normal random variable z follows... ,in The power of additive white Gaussian noise And transmit power constraints and They were 5dB and 38dB respectively;

[0089] The transmitter uses a beamforming matrix for The beamforming matrix used at the receiver is The power allocated to each corresponding stream signal is ,in For each data stream, a transmit beamforming vector is assigned. For each data stream, a receive beamforming vector is assigned. The power value allocated to each data stream;

[0090] The SNR corresponding to the k-th data stream can then be expressed as:

[0091]

[0092] in The power allocated to each data stream, For each data stream, a receive beamforming vector is assigned. For each data stream, a transmit beamforming vector is assigned. For the channel between the transmitter and receiver, the maximum SNR is limited to 30 dB due to power constraints;

[0093] The data rate of the k-th data stream can then be expressed as,

[0094] .

[0095] Furthermore, the multi-stream power allocation process in a large-scale MIMO communication system is optimized using a Markov decision process, specifically as follows:

[0096] The optimization problem under the non-negative power set is defined as maximizing the total spectral efficiency of the system, where all elements in the set satisfy the maximum power constraint, as expressed in the following formula.

[0097]

[0098] The optimization problem is constrained by the minimum transmit power. With maximum transmission power Due to the limitation that the transmit power of each stream signal cannot be less than 0; since the beamforming matrices used by the transmitter and receiver are known, the main focus is on optimizing the power allocation vector to achieve the mapping from the current CSI to the power allocation scheme.

[0099] Further, step S2 includes:

[0100] A deterministic policy gradient network structure is constructed based on the DDPG algorithm, comprising a policy network and a value network. Both the policy network and the value network adopt a dual neural network structure containing an online network and a target network. The online policy network has the same structure as the target policy network, including an input layer, n1 sequential fully connected layers, and an output layer. The output layer of both the online policy network and the target policy network includes K neurons. The online value network has the same structure as the target value network, including an input layer, n2 sequential fully connected layers, and an output layer. The output layer of both the online value network and the target value network includes one neuron.

[0101] like Figure 2 As shown, a power allocation algorithm based on DDPG is constructed, and a dual-network architecture of deterministic policy gradient algorithm is built, including a policy network and a value network. The policy network generates action a based on the state value s, and the value network generates the power value function Q based on the state and action pair (s,a). The output gradient of the Q function is then fed back to the policy network so that the policy network updates the policy in the direction that increases the Q value. This process cleverly combines the approximation of the value function with the policy gradient.

[0102] The hyperparameters of the network architecture are set to [value], and the learning rate of the policy network is set to [value]. ,by The exploration efficiency is used to explore the action space, and the learning rate of the value network is set to... ;

[0103] The fully connected network architecture of the policy network includes an input layer, three fully connected layers, and an output layer. The input layer uses a linear activation function, the fully connected layers use the ReLU activation function and 512 neurons, and the output layer uses the softmax activation function and K neurons for the data stream.

[0104] The fully connected network structure of the value network includes an input layer, two fully connected layers, and an output layer. The input and output layers use linear activation functions, while the fully connected layers use the ReLU activation function and have 256 neurons.

[0105] Further, step S3 includes:

[0106] S31: Define the state space. In the communication system, the state information is set as the SNR of each data stream at the current time plus the power allocation coefficient of the previous time. The current system state matrix is ​​specifically represented as follows:

[0107]

[0108] in Let be the power distribution coefficient at time t-1. The power allocated to the k-th data stream at time t-1. When t=1, ; , Let SNR be the SNR of the k-th data stream at time t; , Let t be the channel from the transmitter to the receiver at time t;

[0109]

[0110] Let be the Rayleigh fading channel at time t;

[0111] S32: The power of a communication system is a non-negative continuous scalar, subject to the maximum power. Due to limitations, the DDPG online policy network receives the state matrix at time t. It directly outputs the corresponding deterministic action. This action is generated by the scaling Softmax function, and the system action in the action space is set as follows:

[0112]

[0113] Where X represents the output of the activation function, which is... The generated real-valued vector, where These are the weights, biases, and activation functions of the current layer, respectively. This is the output of the activation function of the previous layer;

[0114] S33: Environment Interaction: Adaptive Gaussian noise with exploration intensity decreasing with training steps is used.

[0115]

[0116] in, With a mean of 0, the covariance matrix is Multidimensional Gaussian noise, for An identity matrix of order 1. Let be the number of data streams actually transmitted at time t. , The standard deviation is adaptively adjusted during training. It gradually decreases with increasing training steps, linearly decreasing from an initial value of 0.03 and terminating at 0.003-0.005. The formula for the decrease in step size is as follows:

[0117]

[0118] in The initial standard deviation, This is the final standard deviation. The total number of training rounds makes the exploration efficiency higher in the early stage of training and gradually converges in the later stage.

[0119] The training process achieves the exploration of the action space by adding noise to the movements.

[0120]

[0121] in, These are set random parameters during training. The power values ​​allocated to the K data streams at time t+1 are as follows: This means controlling the sum of the power values ​​of the K data streams to the minimum transmit power. With maximum transmission power between;

[0122] The output of the online policy network after exploring the action space is denoted as:

[0123]

[0124] in, It is an online policy network. For parameters of the online policy network, This represents the state matrix received by the online policy network at time t. The output after the exploration behavior in the action space;

[0125] S34: Set the system reward value by adding a series of penalty terms to the average spectral efficiency. The current system reward value is specifically set as follows:

[0126]

[0127] in, , , , The percentage parameters for each reward component are all limited to the range [0,1]. , Let be the number of data streams actually transmitted at time t. , , Represents the time t at time t. The power value allocated to each actual data stream being transmitted; When t=1, .

[0128] Furthermore, the specific training process for step S3 is as follows:

[0129] S35: The online policy network receives the state matrix at time t. Output the exploration behavior through the action space. Further obtain system reward value With the state matrix at the next time step ;

[0130] S36: Composition Sample The samples are stored in the experience replay pool. When the number of samples reaches the capacity threshold of the experience replay pool, the priority of the samples in the experience replay pool is adjusted by using time-series differential error. The priority experience replay mechanism is used to extract the samples of the preset batch with the highest priority according to the sample priority.

[0131] S37: Of the samples drawn ( The input to the online value network is , and the output of the online value network is the corresponding current action value. The online policy network is updated by maximizing the action value in the current state through gradient ascent.

[0132] S38: From the sample As input to the target policy network, refer to steps S32 and S33 to generate the exploration behavior through the action space corresponding to the state at the next time step. ,by( The target value network is input to the target value network, which generates the corresponding target action value. The parameters of the online value network are updated by minimizing the temporal difference error between the predicted action value and the target action value through gradient.

[0133] in, The generation process formula is the same as that of steps S32 and S33, the only difference being that the online policy network is transformed into the target policy network and substituted into the calculation.

[0134] S39: Update the parameters of the target policy network and the target value network online through a soft update mechanism; for the preset batch samples in the current training round, record the maximum system reward value as the reward value of the current round;

[0135] S40: Repeat steps S35-S39 above, iteratively updating the policy network and value network until the total number of training rounds is reached. Then, save the online policy network model weight corresponding to the maximum reward value in each round.

[0136] Furthermore, the step S37 above, which involves updating the online policy network by maximizing the action value in the current state through gradient ascent, specifically involves:

[0137]

[0138]

[0139] in, Let the loss function be the online policy network. For the parameters of the online value network, The number of samples to be drawn in a pre-defined batch in the experience replay pool. To draw a sample set for a pre-defined batch, It is the state matrix at time t. To calculate the gradient of the online policy network parameters, To output the function of the online policy network Find the gradient. Indicates that the online value network is based on The input is the value of the current action, and the output is the corresponding value of the current action. Update according to the following formula:

[0140]

[0141] in, This is the learning rate of the online policy network; repeat step S37 until... convergence;

[0142] In step S38 above, the parameters of the online value network are updated by minimizing the temporal difference error between the predicted action value and the target action value using gradients. Specifically:

[0143]

[0144]

[0145] in, The loss function for the online value network, For the parameters of the target policy network, The parameters of the target value network; To calculate the gradient of the target value network parameters; It is a discount factor. The target policy network is based on The input and output are the exploration behaviors through the action space. Representative target value network according to The input is the target action value, which is then output. Update as follows:

[0146]

[0147] in, It is the learning rate of the online value network; repeat step S38 until... convergence;

[0148] The parameters of the target network are synchronously updated from the online network using a soft update mechanism, specifically as follows:

[0149]

[0150] in, It is a soft update coefficient, and .

[0151] The specific method for step S4 is as follows:

[0152] The state matrix at the current moment Input the data into the saved online policy network model, and the online policy network model will directly output the corresponding deterministic actions. This is denoted as the output power distribution scheme. There are K elements consisting entirely of zeros.

[0153] Furthermore, such as Figure 3 As shown, the power allocation method of this invention is compared with other power allocation algorithms. All encoding schemes are based on a subarray regularized zero-forcing precoding strategy. The five methods are the improved deterministic policy gradient algorithm (Re-DDPG) proposed in this invention, and the compared traditional DDPG algorithm, Weighted Minimum Mean Square Error (WMMSE), equal power allocation (MP), and random power allocation (RP). The solid diamond represents the simulation results of the improved DDPG power allocation scheme proposed in this invention, while the dashed lines represent comparisons with existing power allocation schemes. As can be seen from the figure, the power allocation scheme for large-scale MIMO systems based on deep deterministic policy gradient proposed in this invention outperforms other existing schemes.

[0154] The neural network model of this invention employs offline training and online testing, shifting computational load and overhead to the offline training process, significantly reducing the cost in practical applications. Theoretical analysis and simulation results demonstrate that this invention outperforms traditional power allocation algorithms and existing deep reinforcement learning-based intelligent power allocation algorithms. Under the same power conditions, the improved DDPG algorithm exhibits superior spectral efficiency compared to other traditional algorithms.

[0155] In summary, this invention addresses the shortcomings of traditional power allocation methods, the limitations of existing deep learning-based power allocation methods in terms of performance (which are constrained by traditional algorithms and have room for improvement), and the insufficient interpretability and high model complexity of existing deep reinforcement learning-based methods. By designing and building an intelligent multi-stream power allocation algorithm based on a deterministic gradient algorithm, this invention further improves the spectral efficiency of communication systems.

Claims

1. A power allocation method for large-scale MIMO systems based on deep deterministic policy gradients, characterized in that, Includes the following steps: S1: Construct a model of a large-scale MIMO communication system; S2: Construct a power allocation algorithm based on DDPG and build a deterministic policy gradient network structure; S3: Based on the deterministic policy gradient network structure of S2, the state space and action space of the training network are defined; the deterministic policy gradient network is trained using a reward function with constraints and a priority experience replay mechanism, and the model weight parameters are saved. S4: Load the model weight parameters saved in step S3 and output the power allocation scheme.

2. The power allocation method for large-scale MIMO systems based on deep deterministic policy gradients according to claim 1, characterized in that, Step S1 includes: S11: If the transmitting end is equipped with M antennas and the receiving end is equipped with N antennas, using beamforming technology, and the number of transmitted data streams is K, then the number of data streams is: ； S12: Considering the same time-frequency resources, the channel from the transmitter to the receiver... Modeled as, ； in For a Rayleigh fading channel, g is the large-scale fading coefficient. According to the LTE standard, the large-scale fading coefficient g is modeled as... ； in Let be a log-normal random variable, following the... , It is the distance from the sender to the receiver. The standard deviation is the preset value. S13: The transmitting end uses a beamforming matrix. The receiver uses a beamforming matrix. The power allocated to each corresponding stream signal is ,in For the transmit beamforming vector of the k-th data stream, For the receive beamforming vector of the k-th data stream, The power value allocated to the k-th data stream; ; The SNR corresponding to the k-th data stream can be expressed as: ； in The power of additive white Gaussian noise, ,in For noise power density, Channel bandwidth; The data rate of the k-th data stream can then be expressed as: 。 3. The power allocation method for a large-scale MIMO system based on deep deterministic policy gradient as described in claim 2, characterized in that, The specific method of step S2 includes: A deterministic policy gradient network structure is constructed based on the DDPG algorithm, comprising a policy network and a value network. Both the policy network and the value network adopt a dual neural network structure containing an online network and a target network. The online policy network has the same structure as the target policy network, including an input layer, n1 sequential fully connected layers, and an output layer. The output layer of both the online policy network and the target policy network includes K neurons. The online value network has the same structure as the target value network, including an input layer, n2 sequential fully connected layers, and an output layer. The output layer of both the online value network and the target value network includes one neuron.

4. The power allocation method for a large-scale MIMO system based on deep deterministic policy gradient as described in claim 3, characterized in that, In step S3, based on the deterministic policy gradient network structure of S2, the state space and action space of the training network are defined; the specific method of using a reward function with constraints is as follows: S31: Define the state space. In the communication system, the state information is set as the SNR of each data stream at the current time plus the power allocation coefficient of the previous time. The state matrix at time t is specifically represented as follows: ； in Let be the power distribution coefficient at time t-1. The power allocated to the k-th data stream at time t-1. When t=1, ; , Let SNR be the SNR of the k-th data stream at time t; , Let t be the channel from the transmitter to the receiver at time t; S32: The power of a communication system is a non-negative continuous scalar, subject to the maximum power. Due to limitations, the DDPG online policy network receives the state matrix at time t. It directly outputs the corresponding deterministic action. : ； in This represents the output of the k-th neuron in the output layer of the online policy network. This refers to the k-th action output by the online policy network at time t. S33: Environment Interaction: Adaptive Gaussian noise with exploration intensity decreasing with training steps is used. ； in, With a mean of 0, the covariance matrix is Multidimensional Gaussian noise, for An identity matrix of order 1. Let be the number of data streams actually transmitted at time t. , The standard deviation is adaptively adjusted during training. The standard deviation gradually decreases as the number of training steps increases. The formula for the decrease in step size is expressed as: ； in The initial standard deviation, This is the final standard deviation. The total number of training rounds makes the exploration efficiency higher in the early stage of training and gradually converges in the later stage. The training process achieves the exploration of the action space by adding noise to the movements. ； in, These are set random parameters during training. The power values ​​allocated to the K data streams at time t+1 are as follows: This means controlling the sum of the power values ​​of the K data streams to the minimum transmit power. With maximum transmission power between; The output of the online policy network after exploring the action space is denoted as: ； in, It is an online policy network. For parameters of the online policy network, This represents the state matrix received by the online policy network at time t. The output after the exploration behavior in the action space; S34: Set the system reward value by adding a series of penalty terms to the average spectral efficiency. The current system reward value is specifically set as follows: ； in, , , , The percentage parameters for each reward component are all limited to the range [0,1]. , , Represents the time t at time t. The power value allocated to each actual data stream being transmitted; When t=1, .

5. A power allocation method for a large-scale MIMO system based on deep deterministic policy gradients according to claim 1 or 4, characterized in that, In step S3, a deterministic policy gradient network is trained using a reward function with constraints and a priority experience replay mechanism. The specific method for saving the model weight parameters is as follows: S35: The online policy network receives the state matrix at time t. Output the exploration behavior through the action space. Further obtain system reward value With the state matrix at the next time step ; S36: Composition Sample The samples are stored in the experience replay pool. When the number of samples reaches the capacity threshold of the experience replay pool, the priority of the samples in the experience replay pool is adjusted by using time-series differential error. The priority experience replay mechanism is used to extract the samples of the preset batch with the highest priority according to the sample priority. S37: Of the samples drawn ( The input to the online value network is , and the output of the online value network is the corresponding current action value. The online policy network is updated by maximizing the action value in the current state through gradient ascent. S38: From the sample As input to the target policy network, refer to steps S32 and S33 to generate the exploration behavior through the action space corresponding to the state at the next time step. ,by( The target value network is input to the target value network, which generates the corresponding target action value. The parameters of the online value network are updated by minimizing the temporal difference error between the predicted action value and the target action value through gradient. S39: Update the parameters of the target policy network and the target value network online through a soft update mechanism; for the preset batch samples in the current training round, record the maximum system reward value as the reward value of the current round; S40: Repeat steps S35-S39 above, iteratively updating the policy network and value network until the total number of training rounds is reached. Then, save the online policy network model weight corresponding to the maximum reward value in each round.

6. The power allocation method for a large-scale MIMO system based on deep deterministic policy gradient as described in claim 5, characterized in that, Step S37, which involves updating the online policy network by maximizing the action value in the current state through gradient ascent, specifically includes: ； ； in, Let the loss function be the online policy network. For parameters of the online value network, The number of samples to be drawn in a pre-defined batch in the experience replay pool. To select a sample set for a pre-defined batch, It is the state matrix at time t. To calculate the gradient of the online policy network parameters, To output the function of the online policy network Find the gradient. This indicates the value of the current action corresponding to the output of the online value network. Update according to the following formula: ； in, This is the learning rate of the online policy network; repeat step S37 until... convergence.

7. The power allocation method for a large-scale MIMO system based on deep deterministic policy gradient as described in claim 5, characterized in that, In step S38, the parameters of the online value network are updated by minimizing the temporal difference error between the predicted action value and the target action value using gradients. Specifically: ； ； in, The loss function for the online value network, For the parameters of the target policy network, The parameters of the target value network; To calculate the gradient of the target value network parameters; It is a discount factor. The target policy network output represents the exploration behavior through the action space. The target value network outputs the corresponding target action value. Update as follows: ； in, It is the learning rate of the online value network; repeat step S38 until... convergence.

8. The power allocation method for a large-scale MIMO system based on deep deterministic policy gradient as described in claim 5, characterized in that, In step S39, the parameters of the target network are synchronously updated from the online network using a soft update mechanism, specifically as follows: ； in, It is a soft update coefficient, and ; and These are the parameters for the online policy network and the target policy network, respectively; and These are the parameters for the online value network and the target value network, respectively.

9. The power allocation method for a large-scale MIMO system based on deep deterministic policy gradient as described in claim 4, characterized in that, The specific method for step S4 is as follows: The state matrix at the current moment Input the data into the saved online policy network model, and the online policy network model will directly output the corresponding deterministic actions. This is denoted as the output power distribution scheme. There are K elements consisting entirely of zeros.

Images