Multi-cell downlink MIMO-OFDM system power distribution method

Abstract

The invention discloses a power distribution method for a multi-cell downlink MIMO-OFDM (Multiple Input Multiple Output-Orthogonal Frequency Division Multiplexing) system, which comprises the following steps: firstly, establishing a long-term throughput and power switching cost model for a communication system, and solving by adopting a training execution architecture based on deep reinforcement learning, the architecture being composed of a power decision module and a training center evaluation module; the intelligent agent construction power decision module of the cell is used for outputting a local power distribution scheme; inputting the local accumulated reward value into a QMIX network of a training center by each agent, and fitting a global accumulated reward value; finally, a power decision module and global QMIX network parameters are updated using the global cumulative reward value. By adopting the method provided by the invention, each agent only needs local observation to realize power distribution; in addition, the training center only needs each agent to locally accumulate the reward value instead of global information, so that the signaling overhead can be effectively reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of mobile communications, and in particular relates to a multi-cell downlink MIMO-OFDM system power allocation method. Background Art

[0002] In the past, the traditional convex optimization method was used to solve the wireless resource allocation problem of multi-cell downlink MIMO system, mainly based on centralized architecture and single time slot optimization objectives. In the centralized architecture, the base station collects the channel and service demand information of the access users and uploads it to the training center. The training center is responsible for training the wireless resource allocation method and sending it to the base station for execution. Due to limited computing resources and the high complexity of the centralized convex optimization algorithm, it is difficult to complete millisecond-level calculations. In addition, the uploading and sending of information will also bring great delays; in the single time slot optimization objective problem, the allocation of wireless resources is completed within a time slot, and the main optimization objectives are to maximize throughput, maximize energy efficiency, and minimize transmission power consumption. However, there are fewer studies on multi-time slot optimization problems that are more engineering and practical. In multi-time slot optimization problems, real-time dynamic resource allocation is required according to environmental changes, which puts higher requirements on the real-time and long-term effectiveness of resource allocation schemes. Summary of the invention

[0003] Purpose of the invention: In order to solve the problems existing in the above-mentioned prior art, the present invention provides a power allocation method for a multi-cell downlink MIMO-OFDM system.

[0004] Technical solution: A multi-cell downlink MIMO-OFDM system power allocation method, specifically comprising the following steps:

[0005] Step 1: With the goal of maximizing the downlink capacity of all users in the system within the scheduling period and minimizing the power switching between adjacent scheduling periods, a long-term throughput and power switching cost model A1 is constructed, and the maximum rated transmit power P0 of each antenna and the power difference threshold of adjacent scheduling periods are used. As a constraint;

[0006] Step 2: Construct the power decision module and training center evaluation module of the MIMO-OFDM system, and use deep reinforcement learning with centralized training and distributed execution to solve model A1, so as to obtain the power allocation solution of the MIMO-OFDM system.

[0007] Furthermore, the expression of the long-term throughput and power switching cost model A1 in step 1 is:

[0008]

[0009] Where L represents the total number of base stations, K represents the total number of users, t represents the t-th scheduling period, T represents the total number of scheduling periods, p(t) represents the power allocation matrix in the t-th scheduling period, p(t) = (p1(t), ..., p l (t),…,p L (t)), pl(t) represents the power allocation matrix of base station l, represents the power allocation matrix allocated by base station l to user k in the tth scheduling cycle, Subscript M represents the total number of antennas of base station l, and N represents the total number of subcarriers of base station l; if It means that user k is connected to the nth subcarrier of base station l; if This means that user k has no connection with the nth subcarrier of base station l; represents the channel gain from the mth antenna of base station l to the kth user on the nth subcarrier in the tth scheduling period; for The initial value of A is the antenna gain, c is the speed of light, and f c is the frequency of the band, is the distance from base station l to user k, q is the path loss factor, It is the small-scale fading from the nth subcarrier of the mth antenna of base station l to the kth user in the initial scheduling period; represents small-scale fading, ρ represents the correlation factor, w(t) represents the Gaussian white noise process; R(p(t), t) represents the total downlink capacity of all users in the MIMO-OFDM system in the t-th scheduling period; λ is the cost factor of power switching, χ(p(t), t) represents the difference in power allocation of the MIMO-OFDM system in the t-th scheduling period and the previous adjacent scheduling period; the expressions of R(p(t), t) and χ(p(t), t) are as follows:

[0010]

[0011]

[0012] Where B represents the total bandwidth of the base station. represents the signal-to-interference-noise ratio of the kth user on the nth subcarrier of the base station connected to it in the tth scheduling period, The expression is as follows:

[0013]

[0014] in, is the white noise variance.

[0015] Furthermore, the power decision module includes a local power decision network, a training center strategy network having a structure consistent with the local power decision network, a training center splicing network and a training center weight adjustment network;

[0016] The local power decision network selects a local action according to the local observation of the base station; the training center strategy network is used to update the parameters of the network local power decision network, the training center splicing network is used to splice the strategies output by the training center strategy network during training, and calculate the joint action entropy, thereby outputting the joint strategy and the joint action entropy; the training center weight adjustment network is used to adjust the weight of the training center strategy network;

[0017] The training center evaluation module adopts a QMIX network, including a source Q network, a target Q network, a source mixed network and a target mixed network.

[0018] Furthermore, the step 2 is specifically as follows:

[0019] Step 2.1: Initialize the subcarrier allocation vector x, where x is The constructed matrix obtains the initial power distribution state according to the average power distribution;

[0020] Step 2.2: Based on the local observation o of the tth scheduling period l (t) to obtain the global state s(t), the expression of s(t) is: s(t) = (o 1 (t),o 2 (t),…,o M (t)), o l (t) = {H l (t),R l (t-1),u l (t-1)}, H l (t) is the channel gain matrix from base station l to its connected users in the tth scheduling period, R l (t-1) is the downlink transmission rate from base station l to the users of this base station in the t-1th scheduling period; the local observation rate of base station l in the tth scheduling period is l (t) is input to the local power decision network, which selects action u through a random strategy. l (t),u l The expression of (t) is:

[0021]

[0022] in, is the mean value of the output action, is the variance of the output action, It follows a Gaussian distribution with a mean of 0 and a variance of 1.

[0023] Step 2.3: Set the local action u l (t) is applied to the MIMO-OFDM system and model A1 is used as the optimization strategy to calculate the local reward r of base station l. l (t);

[0024] Step 2.4: Substitute s(t+1), u l (t), r l (t), o l (t) is combined into the local sample point e corresponding to the tth scheduling period l (t);

[0025] Step 2.5: Set each local sample point at a delay of T u After a scheduling cycle, it is transmitted to the sample pool as a global sample point. When the number of sample pools is greater than the preset minimum number of training sample points D, the power decision module and the training center evaluation module are trained, specifically:

[0026] Step 2.5.1: Randomly select D samples from the sample pool to form a training set, and use the training set to train the source Q network, the source hybrid network, the training center strategy network, and the training center weight adjustment network; and update the parameters of these networks;

[0027] Step 2.5.2: According to the training results of step 2.5.1, update the parameters of the target Q network and the parameters of the target hybrid network; every T d In each scheduling cycle, the local power decision network parameters are updated according to the parameters of the training center strategy network;

[0028] Step 2.6: If t￡T, enter the next scheduling cycle and go to step 2.2, otherwise go to step 2.7, where T represents the total number of scheduling cycles;

[0029] Step 2.7: Determine whether the training round E has reached the total training round. If so, go to step 2.8. Otherwise, add 1 to the training round and go to step 2.1.

[0030] Step 2.8: The local power decision network makes a decision based on local observations and obtains a power allocation plan.

[0031] Further, the subcarrier allocation vector x is initialized in step 2.1 as follows: Select the user at the initial moment Connected to base station l, k1 is the set Users in is the set of users connected to base station 1, Represents the total number of users connected to base station 1.

[0032] Furthermore, in step 2.2, R l The expression for (t-1) is:

[0033]

[0034] in, is the set of users connected to base station 1, represents the total number of users connected to base station l, k1 is the set Users in .

[0035] Furthermore, the loss function L(θ c ), training the central policy network L(φ c ) and the loss function for training the central weight-adjusted network The expression is as follows:

[0036]

[0037] Among them, t d represents the time slot of the dth sample, γ is the long-term cumulative reward decay factor, and u c (t d ) is the joint action output by the splicing strategy network, Z(u c (t d )) is the joint action entropy output by the splicing strategy network, Q′(s(t d )),

[0038] Q tot (s(t d ),u(t d )) is as follows:

[0039]

[0040] Among them, s(t d ) represents the global state of the MIMO-OFDM system, represents the output of the target Q network corresponding to base station l, represents the target hybrid network, Q tot (.) represents the source hybrid network; α is the weight,

[0041] represents the output of the source Q network corresponding to base station l, is the target entropy.

[0042] Beneficial effects: The present invention first establishes a long-term throughput and power switching cost model for the communication system, and solves it using a training execution architecture based on deep reinforcement learning. The architecture consists of a power decision module and a training center evaluation module. First, the agent of the cell constructs a power decision module to output a local power allocation plan. Subsequently, each agent inputs the local cumulative reward value into the QMIX network of the training center to fit a global cumulative reward value. Finally, the global cumulative reward value is used to update the power decision module and the global QMIX network parameters. Using the method of the present invention, each agent only needs local observations to achieve power allocation. In addition, the training center only needs the local cumulative reward value of each agent instead of global information, which can effectively reduce signaling overhead. The present invention has lower local computational complexity and information interaction delay. At the same time, it can take into account both downlink throughput and the cost of power switching, and has better performance. DETAILED DESCRIPTION

[0043] The exemplary embodiments of the present invention and their description are used to explain the present invention and do not constitute improper limitations of the present invention.

[0044] The present invention provides a power allocation method for a multi-cell downlink MIMO-OFDM system. The method is for a multi-cell downlink MIMO-OFDM wireless communication system, and takes a long-term throughput and power switching cost compromise function as an objective function to optimize the power allocation scheme of multiple time slots. First, a long-term throughput and power switching cost model is established. In the model, each time slot base station can only observe the channel status, power allocation scheme and other information of the current time slot system, and cannot obtain the channel status, power allocation scheme and other information of future time slots. For the optimization model in which the multi-time slot information is locally known, a training execution architecture based on deep reinforcement learning is used for solving. The architecture can avoid the huge computing overhead required by the centralized architecture, and has lower local computing power and information interaction delay; at the same time, it can take into account the downlink throughput and the cost of power switching, and has better performance.

[0045] In order to better illustrate the method of the present invention, it is described below in conjunction with more detailed examples:

[0046] In a multi-cell MIMO-OFDM downlink communication system consisting of L cells and K users, there is a base station in each cell for signal transmission and reception. Each base station is also regarded as an intelligent agent, interacting with the environment during the reinforcement learning process. Each base station is equipped with M isotropic omnidirectional transmitting antennas, and the rated transmission power of each antenna of the base station is P0. K single-antenna users are randomly distributed in the L cells. Any user is only connected to one base station at a time. The connection relationship between the user and the base station is determined when the system is generated and will not be changed. The base station set is The set of users connected to base station 1 is this The user serial number corresponding to a user in the entire system is

[0047] In this co-frequency multi-cell cellular system, each base station operates in the same frequency bandwidth. The total bandwidth of the frequency band is B, which is divided into N orthogonal subcarriers. The bandwidth of each orthogonal subcarrier is B / N. Each subcarrier can only be assigned to one user at a time, and a user can connect to multiple subcarriers at the same time. The connection relationship between the subcarrier and the user is determined at the initial scheduling time and will not be changed. is the connection matrix between users and subcarriers. Indicates that user k is connected to the nth subcarrier of base station l, otherwise

[0048] The first step is to establish a long-term throughput and power switching cost model.

[0049] use It represents the channel gain from the mth antenna of base station l to the kth user on the nth subcarrier within the scheduling period t. The scheduling period refers to the length of time that the channels of all users remain relatively stable. The duration of each scheduling period is one time slot. represents the channel gain from the mth antenna of base station l to the kth user on the nth subcarrier in the initial time slot, The expression is:

[0050]

[0051] Where A is the antenna gain, c is the speed of light, and f c is the frequency of the band, is the distance from base station l to user k, q is the path loss factor, It is the small-scale fading from the mth antenna of base station l to the kth user on the nth subcarrier during the initial scheduling period.

[0052] A first-order autoregressive model is used to describe the small-scale fading of two consecutive scheduling cycles. The correlation between Modeled as:

[0053]

[0054] Where ρ represents the correlation factor between small-scale fading at two consecutive sampling moments, 0≤ρ≤1, and w(t) is a Gaussian white noise process that satisfies the distribution in is the white noise variance.

[0055] Define the channel gain matrix between base station l and user k in scheduling period t The channel gain matrix H from base station l to its connected users l (t), the channel gain matrix H(t) of the system is:

[0056]

[0057] in, Represents the set of real numbers.

[0058] remember is the power allocated to the kth user by the mth antenna of base station l on the nth subcarrier in scheduling period t, Since each antenna has a maximum rated transmit power P0, The following constraints are met:

[0059]

[0060] In addition, if user k is not connected to the nth subcarrier of base station l, then in the scheduling period t, the power allocated by the mth antenna of base station l to the kth user on the nth subcarrier is 0, that is, That is, the following constraints are satisfied:

[0061]

[0062] Considering that frequent power switching in the existing network will lead to an increase in hardware power consumption, Add the following constraints to limit the switching range of power in adjacent scheduling cycles:

[0063]

[0064] in, is the power difference threshold between adjacent scheduling periods, and |·| represents the absolute value.

[0065] Define χ(p(t), t) as the difference between the system power allocation in scheduling period t and its previous adjacent scheduling period, expressed as:

[0066]

[0067] Define the power allocation matrix allocated by base station l to user k in scheduling period t The power allocation matrix p of base station l l (t), the power allocation matrix p(t) of the system is:

[0068]

[0069] in, Represents the set of real numbers.

[0070] make is the channel reception scalar of the k-th user on the n-th subcarrier of the base station connected to it in scheduling period t, which is expressed as follows:

[0071]

[0072] in, is the signal sent by the mth antenna of the lth base station to the nth subcarrier in the scheduling period t to the user k. They are all independent and identically distributed random variables with a mean of 0 and a variance of 1. is Gaussian white noise, in is the white noise variance.

[0073] The signal-to-interference-to-noise ratio of the kth user on the nth subcarrier of the base station connected to it in the scheduling period t is but The following are expressed:

[0074]

[0075] Therefore, the downlink capacity R obtained by the kth user in scheduling period t is k for:

[0076]

[0077] The total downlink capacity for all users in the system is:

[0078]

[0079] The optimization goal is to maximize the downlink capacity of all users in the system within a scheduling period, while also minimizing the power switching between adjacent scheduling periods. Therefore, this is a non-convex problem with multiple scheduling periods and dual objectives. The problem is specifically stated as follows:

[0080]

[0081] Where T is the number of scheduling cycles for power optimization, λ≥0 is the cost factor of power switching, It means that the optimization target y is maximized by optimizing the variable s.

[0082] In the second step, the solution process is as follows: build the system's power decision module and training center evaluation module to solve the problem using deep reinforcement learning with distributed execution of centralized training.

[0083] First, a power decision module is constructed, wherein the power decision module includes a local power decision network, a training center strategy network, a training center splicing network, and a training center weight adjustment network;

[0084] The local power decision network is expressed as The training center policy network is expressed as The training center strategy network is consistent with the local power decision network structure, where: and are the network parameters of the local power decision network of agent l (i.e., base station) and the training center strategy network; the two outputs of the training center splicing network are respectively expressed as and Z(u c (t)), is a joint action, Z(u c (t)) is the joint action entropy; the training center weight adjustment network is α is the weight, where Adjust the network parameters of the network for training the central weights, o l (t) is the local observation of agent l in scheduling period t, and its expression is:

[0085] o l (t) = {H l (t),R l (t-1),u l (t-1)}

[0086] Among them, H l (t) is the channel gain matrix from base station l to its connected users in scheduling period t, and its expression is:

[0087]

[0088]

[0089] is a complex set; R l (t-1) is the downlink transmission rate from base station l to the users of this base station in the scheduling period t-1, and the specific expression is:

[0090]

[0091] in, is the set of users connected to base station 1, represents the total number of users connected to base station l, k1 is the set Users in the collection The corresponding user number in the system is u l (t-1) is the local action of base station l in scheduling period t-1, and the specific expression is:

[0092]

[0093] in, The global action can be expressed as u(t) = {u 1 (t),u 2 (t),…,u L (t)}.

[0094] Secondly, a training center evaluation module network is constructed, wherein the evaluation module includes a source Q network, a target Q network, a source hybrid network and a target hybrid network.

[0095] The source Q network is represented as The target Q network is expressed as The inputs of the source Q network and the target Q network are local states (i.e., local observations) and local action outputs. The source hybrid network is represented as The target hybrid network is expressed as in, and are the network parameters of the source Q network and the target Q network, θ tot and are the network parameters of the source hybrid network and the target hybrid network respectively, s(t) is the global state of the scheduling period t, which is composed of the local observations of each agent and is expressed as:

[0096] s(t)=(o 1 (t),o 2 (t),…,o M (t))

[0097] The source Q network and the target Q network have the same structure, and the source hybrid network and the target hybrid network have the same network structure.

[0098] Next, randomly initialize the network parameters of the training center policy network Source Q network parameters and source mixing network parameters θ tot ; Let each base station local power decision network parameter Target Q network parameters for training center Target hybrid network parameters Initialize the training round E=0.

[0099] In the third step, agent l observes o locally. l (t), output local action u l (t), the reward r for system feedback l (t), the specific steps are as follows:

[0100] Step 1-1: Initialize the subcarrier allocation vector x, averagely allocate power to obtain the initial power allocation state p(-1), observe and obtain the initial global state s(0), and the subcarrier allocation vector x is determined by the following steps:

[0101] For carrier n=1,2,…,N, when user number is of Select a user from among users The expression is

[0102]

[0103] Among them, ||·||2 represents the modulus of the complex number, It means that the variable y is minimized by optimizing x. Represents the corresponding x value when y is maximized; that is, for a specific carrier of a specific base station, the corresponding maximum value user with the worst channel condition on different antennas is selected as its connected user, thereby determining the subcarrier allocation vector x; initialize t=0.

[0104] Step 1-2: Agent l observes local o l (t) is input into the local power decision network, and the network selects action u through a random strategy l (t),u l The expression of (t) is:

[0105]

[0106] in, is the mean value of the output action, is the variance of the output action; It follows a Gaussian distribution with a mean of 0 and a variance of 1.

[0107] Step 1-3: Each agent takes its local action u l (t) acts on the environment, and the environment calculates the local reward r l (t), r l The expression of (t) is:

[0108]

[0109] Step 1-4: The system enters the scheduling period t+1, obtains the global state s(t+1), and each base station obtains the local observation o of the scheduling period t+1 l (t+1), base station l obtains a local sample point e l (t), e l (t) is expressed as:

[0110] e l (t) = {o l (t),ul (t),r l (t),o l (t+1)}.

[0111] Step 1-5: Each local sample point passes through the transmission network with a delay of T u After a scheduling cycle, it is transmitted to the training center to form the global sample point e(tT u ), whose expression is:

[0112] e(tT u )=(s(tT u ),u(tT u ),r(tT u ),s(tT u +1))

[0113] Among them, r(tT u )=∑ l r l (tT u ), store the global sample point into the sample pool; when the number of sample points in the sample pool is greater than the minimum number of training sample points D, proceed to the fourth step.

[0114] The fourth step is to update the parameters of the network in the power decision module and the training center evaluation module of the system. The specific steps are as follows:

[0115] Step 2-1: The training center randomly selects D sample points from the experience playback pool to form a training sample point set remember is the set of network parameters that need to be updated for training the central source Q network, record The set of parameters that need to be updated to train the central policy network.

[0116] Step 2-2: Update the training center source Q network parameters according to the following formula Train the central source mixing network parameters θ tot , training center strategy network parameters and training center weights to adjust network parameters

[0117]

[0118] Among them, ← represents the assignment operation; is the gradient operation; Q is the Q network learning rate, λ π is the policy network learning rate, λ α To adjust the network learning rate for weights, L(θ c ) is the loss function for training the central source Q network, and its expression is

[0119]

[0120] Among them, γ is the long-term cumulative reward decay factor, u c is the output corresponding to the splicing strategy network, namely:

[0121]

[0122] Q′(s(t d ))The specific expression is as follows:

[0123]

[0124] Q tot (s(t d ),u(t d ))The specific expression is as follows:

[0125]

[0126] L(φ c ) is the loss function for training the central policy network, and its expression is:

[0127]

[0128] The loss function of the training center weight adjustment network is expressed as:

[0129]

[0130] in, is the target entropy, that is, the target value that the average entropy should approach. is the action space, express The dimension size of t d represents the time slot of the d-th sample.

[0131] Step 2-3: According to the training results of the training center, soft update the target Q network parameters of the training center and target hybrid network parameters

[0132]

[0133] Among them, τ is the soft update parameter of the target network.

[0134] Step 2-4: Every T d The local strategy network parameters of each base station are updated in a scheduling cycle.

[0135] Step 2-5: If t￡T, set t=t+1 and return to step 1-2; otherwise E=E+1.

[0136] Step 2-6: If E is less than the total number of training rounds N ep , return to step 1-1; otherwise, execute step 5.

[0137] Step 5: Agent l makes a decision based on the initial system state Then the system moves to the next scheduling cycle based on this decision; the action decision output of agent l in scheduling cycle t is The final system action decision output is p * =(p(0),…,p(t),…,p(T-1)), where p(t) is obtained by sparsifying the corresponding u(t).

[0138] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

Claims

1. A multi-cell downlink MIMO-OFDM system power allocation method, characterized in that: The specific steps include: Step 1: With the goal of maximizing the downlink capacity of all users in the system within the scheduling period and minimizing the power switching between adjacent scheduling periods, a long-term throughput and power switching cost model A1 is constructed, and the maximum rated transmit power P0 of each antenna and the power difference threshold of adjacent scheduling periods are used. As a constraint; Step 2: Construct the power decision module and training center evaluation module of the MIMO-OFDM system, and use deep reinforcement learning with centralized training and distributed execution to solve model A1, so as to obtain the power allocation solution of the MIMO-OFDM system.

2. A multi-cell downlink MIMO-OFDM system power allocation method according to claim 1, characterized in that: The expression of the long-term throughput and power switching cost model A1 in step 1 is: Where L represents the total number of base stations, K represents the total number of users, t represents the t-th scheduling period, T represents the total number of scheduling periods, p(t) represents the power allocation matrix in the t-th scheduling period, p(t) = (p1(t), ..., p l (t),…,p L (t)), pl(t) represents the power allocation matrix of base station l, represents the power allocation matrix allocated by base station l to user k in the tth scheduling cycle, Subscript M represents the total number of antennas of base station l, and N represents the total number of subcarriers of base station l; if It means that user k is connected to the nth subcarrier of base station l; if This means that user k has no connection with the nth subcarrier of base station l; represents the channel gain from the mth antenna of base station l to the kth user on the nth subcarrier in the tth scheduling period; for The initial value of A is the antenna gain, c is the speed of light, and f c is the frequency of the band, is the distance from base station l to user k, q is the path loss factor, It is the small-scale fading from the nth subcarrier of the mth antenna of base station l to the kth user in the initial scheduling period; represents small-scale fading, ρ represents the correlation factor, w(t) represents the Gaussian white noise process; R(p(t), t) represents the total downlink capacity of all users in the MIMO-OFDM system in the t-th scheduling period; λ is the cost factor of power switching, χ(p(t), t) represents the difference in power allocation of the MIMO-OFDM system in the t-th scheduling period and the previous adjacent scheduling period; the expressions of R(p(t), t) and χ(p(t), t) are as follows: Where B represents the total bandwidth of the base station. represents the signal-to-interference-noise ratio of the kth user on the nth subcarrier of the base station connected to it in the tth scheduling period, The expression is as follows: in, is the white noise variance.

3. A multi-cell downlink MIMO-OFDM system power allocation method according to claim 2, characterized in that: The power decision module includes a local power decision network, a training center strategy network consistent with the local power decision network structure, a training center splicing network and a training center weight adjustment network; The local power decision network selects a local action according to the local observation of the base station; the training center strategy network is used to update the parameters of the network local power decision network, the training center splicing network is used to splice the strategies output by the training center strategy network during training, and calculate the joint action entropy, thereby outputting the joint strategy and the joint action entropy; the training center weight adjustment network is used to adjust the weight of the training center strategy network; The training center evaluation module adopts a QMIX network, including a source Q network, a target Q network, a source mixed network and a target mixed network.

4. A multi-cell downlink MIMO-OFDM system power allocation method according to claim 3, characterized in that: The step 2 is specifically as follows: Step 2.1: Initialize the subcarrier allocation vector x, where x is The constructed matrix obtains the initial power distribution state according to the average power distribution; Step 2.2: Based on the local observation o of the tth scheduling period l (t) to obtain the global state s(t), the expression of s(t) is: s(t) = (o 1 (t),o 2 (t),…,o M (t)), o l (t) = {H l (t),R l (t-1),u l (t-1)}, H l (t) is the channel gain matrix from base station l to its connected users in the tth scheduling period, R l (t-1) is the downlink transmission rate from base station l to the users of this base station in the t-1th scheduling period; the local observation rate of base station l in the tth scheduling period is l (t) is input to the local power decision network, which selects action u through a random strategy. l (t),u l The expression of (t) is: in, is the mean value of the output action, is the variance of the output action, It follows a Gaussian distribution with a mean of 0 and a variance of 1. Step 2.3: Set the local action u l (t) is applied to the MIMO-OFDM system and model A1 is used as the optimization strategy to calculate the local reward r of base station l. l (t); Step 2.4: Substitute s(t+1), u l (t), r l (t), o l (t) is combined into the local sample point e corresponding to the tth scheduling period l (t); Step 2.5: Set each local sample point at a delay of T u After a scheduling cycle, it is transmitted to the sample pool as a global sample point. When the number of sample pools is greater than the preset minimum number of training sample points D, the power decision module and the training center evaluation module are trained, specifically: Step 2.5.1: Randomly select D samples from the sample pool to form a training set, and use the training set to train the source Q network, the source hybrid network, the training center strategy network, and the training center weight adjustment network; and update the parameters of these networks; Step 2.5.2: According to the training results of step 2.5.1, update the parameters of the target Q network and the parameters of the target hybrid network; every T d In each scheduling cycle, the local power decision network parameters are updated according to the parameters of the training center strategy network; Step 2.6: If t￡T, enter the next scheduling cycle and go to step 2.2, otherwise go to step 2.7, where T represents the total number of scheduling cycles; Step 2.7: Determine whether the training round E has reached the total training round. If so, go to step 2.

8. Otherwise, add 1 to the training round and go to step 2.

1. Step 2.8: The local power decision network makes a decision based on local observations and obtains a power allocation plan.

5. A multi-cell downlink MIMO-OFDM system power allocation method according to claim 4, characterized in that: Initializing the subcarrier allocation vector x in step 2.1 is specifically as follows: Select the user at the initial moment Connected to base station l, k1 is the set The user in U l is the set of users connected to base station 1, Represents the total number of users connected to base station 1.

6. A multi-cell downlink MIMO-OFDM system power allocation method according to claim 4, characterized in that: Step 2.2 R l The expression for (t-1) is: in, is the set of users connected to base station 1, represents the total number of users connected to base station l, k1 is the set Users in .

7. A multi-cell downlink MIMO-OFDM system power allocation method according to claim 4, characterized in that: The loss function L(θ) of the source Q network in step 2.5.1 c ), training the central policy network L(φ c ) and the loss function for training the central weight-adjusted network The expression is as follows: Among them, t d represents the time slot of the dth sample, γ is the long-term cumulative reward decay factor, and u c (t d ) is the joint action output by the splicing strategy network, Z(u c (t d )) is the joint action entropy output by the splicing strategy network, Q′(s(t d )), Q tot (s(t d ),u(t d )) is as follows: Among them, s(t d ) represents the global state of the MIMO-OFDM system, represents the output of the target Q network corresponding to base station l, represents the target hybrid network, Q tot (.) represents the source hybrid network; α is the weight, represents the output of the source Q network corresponding to base station l, is the target entropy.

Images