Multi-cell and multi-service resource allocation method based on multi-agent deep reinforcement learning

Abstract

The present invention is a multi-cell multi-service resource allocation method based on multi-agent deep reinforcement learning, which is applicable to the resource allocation problem of multi-cell eMBB and URLLC user systems. It includes the following steps: Step 1: Construct a multi-agent network for solving the resource allocation of multi-cell eMBB and URLLC user systems; Step 2: State acquisition; Step 3: Subchannel allocation and power allocation; Step 4: Feedback acquisition and parameter update; Step 5: Decision-making drive mechanism. This method effectively reduces the input and output dimensions, signaling overhead and computational complexity of the network, and well guarantees the service satisfaction level of multi-cell eMBB and URLLC users, thereby improving the performance of the entire system.

Description

Technical Field

[0001] The present invention relates to the field of wireless communications, and more specifically to a resource allocation method for processing simultaneous transmission of multi-cell eMBB and URLLC by a multi-agent deep reinforcement learning method, so as to improve the service satisfaction level of eMBB and URLLC users in the multi-cell. Background Art

[0002] The 6G network will be a fully connected world integrating ground wireless and satellite communications. With the support of technologies such as the global satellite positioning system and telecommunications satellite system, it can flexibly adapt to services in different application scenarios such as enhanced mobile broadband (eMBB) and ultra-reliable low latency communications (URLLC). Immersive cloud XR, holographic communications, sensory interconnection and other business applications in 6G have put forward higher requirements for eMBB and URLLC. How to use limited system resources to meet the different needs of the two services has become a key issue in wireless communication networks. Therefore, solving the resource allocation problem under the coexistence of eMBB and URLLC has become a research focus.

[0003] After searching, it was found that X. Wang et al. published a paper entitled "Joint Scheduling of URLLC and eMBB Traffic in 5G Wireless Networks" in IEEE Conference on Computer Communications, pp. 1970-1978, April 2018. The paper proposed a linear model, a convex model and a threshold model to evaluate the loss of eMBB data rate, and to coordinate and optimize the bandwidth allocation of eMBB users and the location of resource grabbing of URLLC traffic under the condition of stable arrival of URLLC traffic. However, in practical applications, URLLC traffic is time-varying, and a long-term optimal solution cannot be obtained using this method. And as the number of users continues to increase and the scale of the system continues to expand, this method has problems such as long periodicity and high computational complexity, so researchers consider applying reinforcement learning methods with strong computing power and learning rate to the problem of wireless network resource allocation.

[0004] After searching the patent, it was found that CN109561504A discloses a resource reuse of URLLC and eMBB based on deep reinforcement learning. The invention first collects the data packet information, channel information and queue information of URLLC and eMBB of M mini-slots as training data; then establishes a URLLC and eMBB resource reuse model based on deep reinforcement learning, and uses the training data to train the model parameters; after the training is completed, the URLLC and eMBB data packet information, channel information and queue information of the current mini-slot are input into the trained model, and finally the resource reuse decision result is obtained to achieve the reasonable allocation and utilization of time-frequency resources and power. However, the invention only considers the resource allocation scheme of single-cell eMBB and URLLC systems. In actual application scenarios, since each cell occupies the same spectrum resources, users in the cell will inevitably be interfered with by its neighboring cells. Therefore, the reasonable allocation of sub-channels and power of multi-cell eMBB and URLLC user systems to improve system performance has become a current research hotspot. Summary of the invention

[0005] The present invention proposes a multi-cell eMBB and URLLC user system resource allocation method based on multi-agent deep reinforcement learning, which solves the problem of multi-cell eMBB and URLLC user system resource allocation based on a multi-agent deep reinforcement learning method, and uses a mode of centralized training and distributed execution of multiple agents to perform global control and reduce the dimension of complex tasks, thereby effectively improving the performance of the system while reducing time costs. Specifically, based on the joint competitive deep Q network (DDQN) and the deep deterministic policy gradient network (DDPG), the subchannel and power allocation schemes of each cell are output respectively, and then the allocation strategy is adjusted according to the feedback of the system to maximize the service satisfaction level of multi-cell eMBB and URLLC users.

[0006] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0007] The present invention is a multi-cell multi-service resource allocation method based on multi-agent deep reinforcement learning, which is applicable to the resource allocation problem of multi-cell eMBB and URLLC user systems. The multi-cell multi-service resource allocation method comprises the following steps:

[0008] Step 1: Build a multi-agent network to solve the resource allocation of multi-cell eMBB and URLLC user systems.

[0009] Specifically, it is assumed that there are N base stations in the multi-cell eMBB and URLLC user system, and each base station has a total of M users randomly distributed in the cell, of which there are B eMBB users and U URLLC users, and M = B + U. Each user is assigned an antenna for receiving and sending data, and there are L subchannels in each base station. Then, different durations are used for transmission according to the user's specific requirements. This patent divides the time domain into 1 millisecond time slots for transmitting eMBB traffic, and further divides the time slots into 7 small time slots for transmitting URLLC traffic. Among them, each time slot has D u URLLC packets arrive, each of size Z u Bytes. Assume that the total bandwidth of the multi-cell system is 3MHz. In order to maximize the service satisfaction level of eMBB and URLLC users under limited spectrum resources, this patent constructs a multi-agent deep reinforcement learning network to solve the sub-channel and power allocation problems of multi-cell eMBB and URLLC users. First, N Q-DNNs and N actor DNNs are established locally, and the local network outputs local sub-channel allocation actions and power allocation actions based on local channel state information. Then, based on DDQN and DDPG, a centralized training network is established in the center, and the network parameters are updated through rewards fed back by the environment, and then the parameters of the local network are updated.

[0010] Step 2: Status acquisition: The channel gain information of eMBB and URLLC users in the cell on different sub-channels of different base stations is used as the current status of the cell. t ; For example, the state of the nth base station at time t is:

[0011]

[0012] Step 3: Subchannel allocation and power allocation: The local neural network takes the state obtained in step 2 as input, and then outputs the local subchannel allocation action and power allocation action. For example, the subchannel and power allocation actions of the nth base station at time t are:

[0013]

[0014] and

[0015]

[0016] Specifically, at the beginning of each time slot, the local state s is obtained n (t) is sent to the corresponding local Q-DNN n′ network and actor DNN n′ network. The local Q-DNN n′ network uses the ∈-greedy strategy to select an action from the local subchannel allocation action space. As the subchannel allocation scheme in the current time slot. Among them, the ∈-greedy strategy refers to randomly selecting an action from the subchannel allocation action space with probability ∈ as Or choose the action with the maximum estimated Q value with probability 1-∈ To balance the exploration of new actions and the use of known actions. At the same time, the local actor DNN n′ network is also activated, using the same state as input, according to To output the corresponding power allocation action, where μ(s n (t);θ′ n ) is the policy function of the local actor DNN n′ network, θ′ n are the network parameters of the local actor DNN n′, represents a random noise process and follows a normal distribution. Finally, the local network outputs the joint subchannel and power allocation actions as follows:

[0017] a(t)={a1(t), a2(t),...,a N (t)}=

[0018] {[C1(t),P1(t)],[C2(t),P2(t)],...,[C N (t), P N (t)]}.

[0019] Step 4: Feedback acquisition and parameter update.

[0020] Each cell receives the above joint subchannel allocation and power allocation action a n (t), from the current state s n (t) Move to the next state s′ n (t), and gives the local reward r n (t), and then fed back to the local network. The local network continues to collect experience e n ={s n (t), a n (t), r n (t), s′ n (t)} and upload it to the central network. The central network receives After that, the global information {s1(t), s2(t), ..., s N (t), a1(t), a2(t), ..., a N (t), r(t), s′1(t), s′2(t),...,s′ N (t)} is stored in the experience pool D in a first-in-first-out manner, where

[0021] In the central network, a multi-agent network is established based on DDQN and DDPG to update the local network parameters. For the parameter update of the local Q-DNN n′: at time t, a part of the sample data is selected from the experience memory pool, and the network parameters α of the central network Q-DNN n are updated by minimizing the following loss function n and β n ,

[0022]

[0023] in,

[0024] and is the network parameter of the central network target Q-DNN n. Then every X steps, the parameters of the central network Q-DNN n are assigned to the corresponding target Q-DNN network, as shown below:

[0025] and

[0026] Finally, the updated network parameter α n and β n The data is transmitted to the local computer to update the parameters of the Q-DNN n′ network.

[0027] For the parameter update of the local actor DNN n′: at time t, the network parameters δ of the central network critic DNN are updated by minimizing the following loss function using the above sample data,

[0028]

[0029] in,

[0030] y(t)=r(t)+γQ(s′1(t),...,s′ N (t), P′1(t), ..., P′ N (t); δ - ), δ - are the network parameters of the target criticDNN.

[0031] Then, the network parameters of the target critic DNN are updated in a soft-update manner as follows:

[0032] δ-←τ c δ+(1-τ c )δ - , where 0<τ c <<1,τ cRepresents the learning rate of the target critic DNN network. Afterwards, the network parameters of the central network actor DNN n are trained by maximizing the expectation of the global reward. The network parameters of the central network actor DNNn are θ n Updated via:

[0033]

[0034] Similar to the target critic network, the network parameters of the target actor DNN are updated as follows:

[0035] Among them, 0<τ n <<1,τ n represents the learning rate of the target actor DNN n network. Finally, the updated network parameters θ n The data is transmitted to the local machine to update the actor DNN n′ network parameters.

[0036] Step 5: Decision-making driving mechanism.

[0037] The present invention designs a decision-driven mechanism, which monitors the state of the system. When the states of two connected time slots are almost the same, a new round of learning process is triggered. Otherwise, the action output in the previous time slot continues to be used as the optimal resource allocation action for the current time slot.

[0038] Specifically, we set a state error threshold ρ,

[0039] ρ=||s(t)-s(t-1)|| 2 , where s(t) represents the state of the current time slot and s(t-1) represents the state of the previous time slot. By monitoring the current state and then comparing it with the state of the previous time slot, it is decided whether to conduct a new round of learning, as shown below:

[0040]

[0041] Among them, a n (t-1) represents the output action of base station n in the previous time slot, Represents the output action of base station n after a new round of learning.

[0042] The beneficial effects of the present invention are as follows: the method of the present invention designs multiple agents locally and centrally for distributed execution and centralized training based on DDQN and DDPG networks, and the method well solves the sub-channel allocation and power allocation problems of multi-cell eMBB and URLLC user systems, and effectively reduces the input and output dimensions, signaling overhead and computational complexity of the network; compared with ordinary reinforcement learning methods, the service satisfaction level of multi-cell eMBB and URLLC users is improved, thereby improving the performance of the entire network; the method combines a multi-agent deep reinforcement learning network for sub-channel allocation and power allocation to improve the system performance of multi-cell eMBB and URLLC during simultaneous transmission, and maximizes the service satisfaction level of multi-cell eMBB and URLLC users while considering the co-frequency interference between cells. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the multi-cell eMBB and URLLC multiplexing scenario of the present invention.

[0044] Figure 2 It is a block diagram of resource allocation of multi-cell eMBB and URLLC user systems based on multi-agent deep reinforcement learning of the present invention.

[0045] Figure 3 It is a schematic diagram of information interaction between a multi-agent network and a multi-cell system of the present invention.

[0046] Figure 4 It is a comparative schematic diagram of multi-cell eMBB and URLLC user service satisfaction of the present invention.

[0047] Figure 5 It is a schematic diagram comparing the time cost of each execution of the method of the present invention and other methods. DETAILED DESCRIPTION

[0048] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0049] The present invention is a joint sub-channel allocation and power allocation method for a multi-cell eMBB and URLLC user system based on multi-agent deep reinforcement learning.

[0050] Assume that there are N base stations in the multi-cell eMBB and URLLC user system, and each base station has a total of M users randomly distributed in the cell, of which there are B eMBB users and U URLLC users, and M = B + U. Each user is assigned an antenna for receiving and sending data, and there are L subchannels in each base station. Then, different durations are used for transmission according to the user's specific requirements. This patent divides the time domain into 1 millisecond time slots for transmitting eMBB traffic, and further divides the time slots into 7 small time slots for transmitting URLLC traffic. Among them, each time slot has D u URLLC packets arrive, each of size Z u Bytes. Assume that the total bandwidth of the multi-cell system is 3MHz. In order to maximize the service satisfaction level of eMBB and URLLC users under the condition of limited spectrum resources.

[0051] This is achieved through the following steps:

[0052] Step 1: Build a multi-agent network to solve the resource allocation of multi-cell eMBB and URLLC user systems.

[0053] Specifically: N Q-DNNs and N actor DNNs are established locally. The local network outputs local sub-channel allocation actions and power allocation actions based on local channel state information. Then, a centralized training network is established in the center based on DDQN and DDPG. The network parameters are updated through rewards fed back by the environment, and then the parameters of the local network are updated. Finally, the agent achieves maximum rewards through continuous learning.

[0054] Step 2: Based on the interference between adjacent cells in the multi-cell eMBB and URLLC user system, establish the signal-to-noise ratio (SINR) and the obtained data rate for each eMBB user and URLLC user, and set the target reward.

[0055] Specifically, the SINR of the lth subchannel received by the eMBB user b from the base station n in the kth mini-slot is:

[0056]

[0057] The SINR of the lth subchannel received by URLLC user u from base station n in the kth mini-slot is:

[0058]

[0059] in, They represent the channel allocation index of user m, the channel gain in the kth mini-slot, and the transmission power of the lth sub-channel received from base station n in the kth mini-slot respectively, and N0 represents the noise power.

[0060] Then, according to the Shannon formula, the transmission rates of eMBB user b and URLLC user u in the kth minislot of the lth subchannel of base station n are obtained as follows:

[0061] and

[0062] Finally, the rate achieved by all eMBB users in base station n in the tth time slot is obtained:

[0063]

[0064] And the rate achieved by all URLLC users at base station n in the tth time slot:

[0065]

[0066] The objective reward of the present invention is to maximize the service satisfaction level of multi-cell eMBB and URLLC users, and the service satisfaction levels of eMBB and URLLC users at base station n are measured respectively by the following formulas.

[0067] and in, is the minimum rate requirement of all eMBB users in base station n in the tth time slot, is the number of URLLC users arriving at base station n in the tth time slot.

[0068] In order to transform the above multi-objective problem into a single-objective problem, the present invention sets the service satisfaction level of multi-cell eMBB and URLLC users as the target reward. The specific optimization problem is described as follows:

[0069] P1:

[0070] stC1:

[0071] C2:

[0072] Indicates the maximum transmission power of base station n.

[0073] Step 3: Set the state, and use the channel gain information of all users in each cell on different sub-channels as the current state s t , such as the state of the nth base station at time t is:

[0074]

[0075] Step 4: Subchannel allocation and power allocation: The local neural network takes the state obtained in step 3 as input, and then outputs the local subchannel allocation action and power allocation action. For example, the subchannel and power allocation actions of the nth base station at time t are:

[0076]

[0077] and

[0078]

[0079] Specifically, at the beginning of each time slot, the obtained local state sn(t) is sent to the corresponding local Q-DNN n′ network and actor DNN n′ network. The local Q-DNN n′ network adopts the ∈-greedy strategy to select an action from the local subchannel allocation action space. As the subchannel allocation scheme in the current time slot. Among them, the ∈-greedy strategy refers to randomly selecting an action from the subchannel allocation action space with probability ∈ as Or choose the action with the maximum estimated Q value with probability 1-∈ To balance the exploration of new actions and the use of known actions. At the same time, the local actor DNN n′ network is also activated, using the same state as input, according to To output the corresponding power allocation action, where μ(s n (t);θ′ n ) is the policy function of the local actor DNN n′ network, θ′ n are the network parameters of the local actor DNN n′, represents a random noise process and follows a normal distribution. Finally, the local network outputs the joint subchannel and power allocation actions as follows:

[0080] a(t)={a1(t), a2(t),...,a N (t)}=

[0081] {[C1(t),P1(t)],[C2(t),P2(t)],...,[C N (t), P N (t)]}.

[0082] Step 5: Feedback acquisition and parameter update.

[0083] Each cell receives the above joint subchannel allocation and power allocation action a n (t), from the current state s n (t) Move to the next state s′ n (t), and gives the local reward rn (t), and then fed back to the local network. The local network continues to collect experience e n ={s n (t), a n (t), r n (t), s′ n (t)} and upload it to the central network, which receives After that, the global information {s1(t), s2(t), ..., s N (t), a1(t), a2(t), ..., a N (t), r(t), s′1(t), s′2(t),...,s′ N (t)} is stored in the experience pool D in a first-in-first-out manner, where

[0084] In the central network, this patent establishes a multi-agent network based on DDQN and DDPG to update the local network parameters. For the parameter update of the local Q-DNN n′: at time t, select a part of the sample data from the experience memory pool and update the network parameters α of the central network Q-DNN n by minimizing the following loss function n and β n ,

[0085]

[0086] in,

[0087] and is the network parameter of the central network target Q-DNN n, and then every X steps, the parameters of the central network Q-DNN n are assigned to the corresponding target Q-DNN network, as shown below:

[0088] and

[0089] Finally, the updated network parameter α n and β n The data is transmitted to the local computer to update the parameters of the Q-DNN n′ network.

[0090] For the parameter update of the local actor DNN n′: at time t, the network parameters δ of the central network critic DNN are updated by minimizing the following loss function using the above sample data,

[0091]

[0092] in,

[0093] y(t)=r(t)+γQ(s′1(t),...,s′ N (t), P′1(t), ..., P′ N (t); δ - ), δ - are the network parameters of the target criticDNN.

[0094] Then, the network parameters of the target critic DNN are updated in a soft-update manner as follows:

[0095] δ - ←τ c δ+(1-τ c )δ - , where 0<τ c <<1,τ c Represents the learning rate of the target critic DNN network. Afterwards, the network parameters of the central network actor DNN n are trained by maximizing the expectation of the global reward. The network parameters of the central network actor DNNn are θ n Updated via:

[0096]

[0097] Similar to the target critic network, the network parameters of the target actor DNN are updated as follows:

[0098] Among them, 0<τ n <<1,τ n represents the learning rate of the target actor DNN n network. Finally, the updated network parameters θ n The data is transmitted to the local machine to update the actor DNN n′ network parameters.

[0099] Step 6: Decision-making driving mechanism.

[0100] The present invention relates to a decision-driven mechanism. By monitoring the state of the system, when the states of two connected time slots are almost the same, a new round of learning process will be triggered. Otherwise, the action output in the previous time slot will continue to be used as the optimal resource allocation action in the current time slot.

[0101] Specifically: set a state error threshold ρ,

[0102] ρ=||s(t)-s(t-1)|| 2 , where s(t) represents the state of the current time slot, and s(t-1) represents the state of the previous time slot. The decision-driving module monitors the current state and then compares it with the state of the previous time slot to decide whether to conduct a new round of learning, as shown below:

[0103]

[0104] Among them, a n (t-1) represents the output action of base station n in the previous time slot, Represents the output action of base station n after a new round of learning.

[0105] like Figure 1-5 As shown, considering the multi-cell eMBB and URLLC system scenarios, the sub-channel and power allocation scheme of each user is jointly optimized. The main parameters of the simulation scenario of this embodiment are shown in Table 1.

[0106] Table 1 Main parameters of the system

[0107]

[0108] Figure 4 and Figure 5 It is a comparative diagram of the algorithm of the present invention and other methods on the user service satisfaction level of multi-cell eMBB and URLLC and the time cost of each execution. As can be seen from the figure, the system performance obtained by the MADRL and MADRL-DD algorithms proposed in the present invention is slightly lower than the exhaustive method, and much higher than the general reinforcement learning algorithm and the random method. In addition, the performance of the MADRL-DD algorithm is infinitely close to that of the MADRL algorithm. It can be seen that the decision-driven module effectively reduces the time cost and computing overhead while ensuring the user service satisfaction level of multi-cell eMBB and URLLC.

[0109] This method effectively reduces the network's input and output dimensions, signaling overhead, and computational complexity, and well ensures the service satisfaction level of multi-cell eMBB and URLLC users, thereby improving the performance of the entire network.

[0110] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A multi-cell multi-service resource allocation method based on multi-agent deep reinforcement learning is applicable to the resource allocation problem of multi-cell eMBB and URLLC user systems, and is characterized by: The multi-cell multi-service resource allocation method comprises the following steps: Step 1: Build a multi-agent network to solve the resource allocation of multi-cell eMBB and URLLC user systems; Step 2: Status acquisition: The channel gain information of eMBB and URLLC users in the cell on different sub-channels of different base stations is used as the current status of the cell. t ; Step 3: Subchannel allocation and power allocation: The local neural network takes the state obtained in step 2 as input, and then outputs the local subchannel allocation action and power allocation action. The specific steps are as follows: Step 3-1: At the beginning of each time slot, obtain the local state s n (t) is sent to the corresponding local Q-DNN n′ network and actor DNN n′ network, where n is the base station; Step 3-2: The local Q-DNNn′ network uses the ∈-greedy strategy to select an action from the local subchannel allocation action space As a subchannel allocation scheme within the current time slot; Step 3-3: At the same time, the local actor DNN n′ network is activated, using the same state as input, according to To output the corresponding power allocation action, where μ(s n (t);θ′ n ) is the policy function of the local actor DNN n′ network, θ′ n are the network parameters of the local actor DNN n′, represents a random noise process and follows a normal distribution; Step 3-4: Finally, the local network outputs the combined sub-channel and power allocation actions as follows: a(t)={a1(t),a2(t),...,a N (t)}={[C1(t),P1(t)],[C2(t),P2(t)],…,[C N (t),P N (t)]}; Step 4: Feedback acquisition and parameter update: The subchannel allocation action and power allocation action obtained in step 3 are transmitted to the system. The system gives rewards and moves to the next state. Then the local network continues to collect the current state, current action, current reward and next state, and uploads them to the experience memory pool. Then, sample data is extracted from the experience memory pool to train the central network parameters and transmitted to the local network. Step 5: Decision-driven mechanism, by monitoring the state of the system, when the states of two connected time slots are almost the same, a new round of learning process will be triggered, otherwise the action output in the previous time slot will continue to be used as the optimal resource allocation action for the current time slot.

2. According to claim 1, the multi-cell multi-service resource allocation method based on multi-agent deep reinforcement learning is characterized in that: The construction of the multi-agent network in step 1 is specifically as follows: Step 1-1: N Q-DNNs and N actor DNNs are established locally. The local network outputs local sub-channel allocation actions and power allocation actions based on local channel state information. Step 1-2: Establish a centralized training network based on DDQN and DDPG in the center, update the network parameters through the rewards fed back by the environment, and then update the parameters of the local network; Step 1-3: The agent maximizes the reward through continuous learning.

3. According to claim 1, the multi-cell multi-service resource allocation method based on multi-agent deep reinforcement learning is characterized in that: In step 4, the feedback acquisition and parameter update are specifically as follows: Step 4-1: Each cell receives the joint subchannel allocation and power allocation action a in step 3 n (t), from the current state s n (t) Move to the next state s′ n (t), and gives the local reward r n (t), and then fed back to the local network; Step 4-2: Local network continues to collect experience n ={s n (t), a n (t), r n (t), s′ n (t)} and upload it to the central network; Step 4-3: The central network receives After that, the global information {s1(t), s2(t), ..., s N (t), a1(t), a2(t), ..., a N (t), r(t), s′1(t), s′2(t),...,s′ N (t)} is stored in the experience pool D in a first-in-first-out manner, where Step 4-4: In the central network, select a portion of sample data from the experience memory pool, update the central network parameters, and finally transmit the updated network parameters to the local to update the local network parameters.

4. According to claim 3, the multi-cell multi-service resource allocation method based on multi-agent deep reinforcement learning is characterized in that: The step 4-4 is specifically as follows: Step 4-4-1: Parameter update of local Q-DNN n′: At time t, select a portion of sample data from the experience memory pool and update the network parameters α of the central network Q-DNN n by minimizing the following loss function: n and β n , in, and is the network parameter of the central network target Q-DNNn, and then every X steps, the parameters of the central network Q-DNNn are assigned to the corresponding target Q-DNN network, as shown below: and Finally, the updated network parameters α n and β n Download to local machine to update the parameters of Q-DNN n′ network; Step 4-4-2: Parameter update of local actor DNN n′: At time t, the network parameters δ of the central network critic DNN are updated by minimizing the following loss function using the above sample data. in, y(t)=r(t)+γQ(s′1(t),...,s′ N (t), P′1(t),..., P′N(t); δ - ), δ - are the network parameters of the target critic DNN; Step 4-4-3: Update the network parameters of the target critic DNN in a soft-update manner, as shown below: d-←t c d+(1-t c )δw Among them, 0<τ c <<1,τ c Represents the learning rate of the target critic DNN network; Step 4-4-4: Train the network parameters of the central network actor DNN n by maximizing the expectation of the global reward. The network parameters of the central network actor DNN n are θ n Updated via: The network parameters of the target actor DNN are updated as follows: Among them, 0<τ n <<1,τ n represents the learning rate of the target actor DNN n network; Step 4-4-5: Update the network parameters θ n The data is transmitted to the local machine to update the actor DNN n′ network parameters.

5. According to claim 1, the multi-cell multi-service resource allocation method based on multi-agent deep reinforcement learning is characterized in that: In step 5, the decision-making driving mechanism is specifically: Step 5-1: Set a state error threshold ρ, ρ=||s(t)-s(t-1)|| 2 , where s(t) represents the state of the current time slot and s(t-1) represents the state of the previous time slot; Step 5-2: The decision-driving module monitors the current state and compares it with the state of the previous time slot to decide whether to conduct a new round of learning, as shown below: Among them, a n (t-1) represents the output action of base station n in the previous time slot, Represents the output action of base station n after a new round of learning.

Images