A resource allocation method based on transfer reinforcement learning

Abstract

The present invention is a resource allocation method based on transfer reinforcement learning, which is applicable to the hybrid multiple access scenario of uplink multi-cell, including: step 1: building a deep reinforcement learning network for resource allocation; step 2: performing MA-DRL interaction with the environment, in each round of interaction; step 3: performing the training process of MA-DRL, the training will be based on the experience blocks obtained by the interaction in step 2, the experience blocks obtained by each interaction with the environment are stored in the memory bank, and value labels are applied, and experience blocks with larger value labels are selected when extracting memory blocks; step 4: based on the resource allocation network, a transfer learning scheme based on MA-DRL is proposed. The resource allocation method proposed by the present invention aims to maximize the total rate of all users, can effectively improve the total rate of users, and can achieve a faster convergence speed after combining with transfer learning.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and specifically relates to a resource allocation method based on transfer reinforcement learning, which is suitable for hybrid multiple access scenarios of uplink multi-cells. Background Art

[0002] At present, people's demand for data rate is growing exponentially, and a substantial increase in data rate requires the consumption of a large amount of spectrum resources. In traditional communication networks, orthogonal multiple access (OMA) technology is generally used for user multiplexing, while non-orthogonal multiple access (NOMA) allows multiple user terminals to share the same resource unit at the same time, which can significantly improve system throughput and has become one of the key technologies of 5G. In actual systems, the serial interference cancellation (SIC) receiver used by NOMA cannot completely eliminate the interference between NOMA users. When the user channel gain difference is too small, NOMA may not be able to meet the service quality of some services, and may not be able to meet the power requirements of all users. Therefore, a hybrid multiple access system that supports both access methods is proposed. How to choose the appropriate random access method (NOMA or OMA) while ensuring the best resource allocation for all users is the key to improving network access capacity.

[0003] After searching for related work, it was found that B.Liu et al. published an article entitled "Resource Allocation for Energy-Efficient MEC in NOMA-Enabled Massive IoT Networks" in IEEE Journal on Selected Areas in Communications, vol.39, no.4, pp.1015-1027, April 2021. The article proposed a resource allocation scheme for mobile edge computing in NOMA. The scheme is solved by matching and sequential convex programming algorithms, but the scheme does not take into account the complex environmental changes in the communication scenario and cannot guarantee the long-term stability of the scheme. Reinforcement learning in deep learning gradually updates its own network in the interaction with the environment and aims at long-term rewards, so it has great advantages in dynamic resource allocation in communication scenarios.

[0004] After searching, it was found that X. Zhang et al. published an article entitled "Deep-Reinforcement-Learning-Based Mode Selection and Resource Allocation for Cellular V2X Communications" in "IEEE Internet of Things Journal, vol.7, no.7, pp.6380-6391, Jul.2020". This article uses deep reinforcement learning to study the joint optimization problem of transmission mode selection and resource allocation for cellular V2X communications to maximize the total capacity of vehicles to infrastructure users.

[0005] Transfer learning, as another branch of deep learning, is used to transfer knowledge in the source domain to the target domain related to the source domain. It is very helpful to improve the robustness and convergence speed of the model. For the frequently changing environment in the communication scenario, adding transfer learning to resource allocation will further improve the performance of the resource allocation strategy. The combination of transfer learning and reinforcement learning can accelerate the learning process of the reinforcement learning agent, and can use the knowledge trained in the source domain and combine it into the target domain to become an effective resource allocation method. Regarding the work combining transfer learning and reinforcement learning, only a few articles are aimed at the resource allocation goals in communication scenarios. In the cognition of technical personnel in this field, there is no research on combining multi-agent reinforcement learning and transfer learning into hybrid multiple access scenarios. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a resource allocation method based on transfer reinforcement learning, which selects the optimal user scheduling and power allocation scheme by maximizing the total rate of all users.

[0007] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0008] The present invention is a resource allocation method based on transfer reinforcement learning, which is applicable to a hybrid multiple access scenario of uplink multi-cells. The resource allocation method comprises the following steps:

[0009] Step 1: Build a deep reinforcement learning network (MA-DRL) for resource allocation, input the channel gain of the current environment into MA-DRL, and MA-DRL calculates the channel gain difference and determines the round of interaction between the network and the environment based on the channel gain difference;

[0010] Step 2: MA-DRL interacts with the environment. In each round of interaction, all cells obtain the current environmental status. MA-DRL, which is used to allocate power allocation and user scheduling resources, obtains resource allocation actions based on the status. MA-DRL then interacts with the environment based on the obtained resource allocation actions to obtain the current reward and the next environmental status.

[0011] Step 3: Perform the training process of MA-DRL. The training will be carried out based on the experience blocks obtained from the interaction in step 2. Each experience block obtained from the interaction with the environment is stored in the memory bank and a value label is added. When extracting the memory block, the experience block with a larger value label is selected.

[0012] Step 4: Based on the deep reinforcement learning network for resource allocation (MA-DRL), a transfer learning scheme based on MA-DRL (T-DRL) is proposed to enhance the ability of MA-DRL to cope with environmental changes, and the two sub-networks in MA-DRL are migrated separately.

[0013] A further improvement of the present invention is that T-DRL is divided into three migration modes, namely, a migration mode of migrating only the power allocation subnetwork, a migration mode of migrating only the user scheduling subnetwork, and a migration mode of migrating both subnetworks. The first migration mode only migrates the DQN network. This migration mode can be selected when the power resources of the environment do not change much. The second migration mode only migrates the MADDPG network. This migration mode can be selected when the sub-channel resources of the environment do not change much. The third migration mode migrates the DQN network and the MADDPG network at the same time. When the overall environment changes, such as when the channel fading mode of the environment changes, it will affect the resource allocation of the two networks at the same time. At this time, this migration mode can be selected. The T-DRL used in the method of the present invention can freely select a suitable migration mode according to different environmental changes.

[0014] The beneficial effects of the present invention are:

[0015] 1. In previous work, it is difficult to find a method to efficiently obtain the optimal solution by using various algorithms to solve NP-hard problems in order to simultaneously allocate sub-channel resources and power resources. The present invention makes full use of the advantages of deep reinforcement learning, adopts centralized training and distributed execution to build a multi-agent deep reinforcement learning neural network, and effectively reduces the interference between cells and the power consumed by transmission while ensuring the normal communication of each user.

[0016] 2. The present invention continuously improves the system's ability to fit the environment and the ability to allocate resources during training, thereby improving the communication quality of the entire communication system and meeting the requirements of future mobile communication systems;

[0017] 3. Different from the traditional transfer learning method, the present invention can enable the system to select a unique transfer method and a suitable transfer model to prevent the occurrence of negative transfer. In the process of studying the resource allocation method of the multi-cell hybrid multiple access system, transfer learning is used to preserve the experience of the trained deep neural network. When facing a new environment with different distributions, the knowledge from the source task is transferred to the task in the target domain, which helps to effectively utilize the experience samples and thus accelerate the convergence of the learner's task. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a specific flow chart of the resource allocation method of the present invention including the transfer learning part and the reinforcement learning part.

[0019] Figure 2 It is a schematic diagram of the T-DRL network framework of the resource allocation method of the present invention.

[0020] Figure 3 It is a total rate performance diagram obtained by resource allocation of MA-DRL for a hybrid multiple access scenario in an embodiment of the present invention.

[0021] Figure 4 It is a migration performance diagram of T-DRL in an embodiment of the present invention under the change of channel fading mode.

[0022] Figure 5 It is a migration performance diagram of the T-DRL in the embodiment of the present invention when the initial position of the user changes.

[0023] Figure 6 It is a migration performance diagram of T-DRL in an embodiment of the present invention when the user location update range changes. DETAILED DESCRIPTION

[0024] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0025] The present invention proposes a method for allocating resources in an uplink multi-cell hybrid multi-access scenario based on transfer reinforcement learning. The technical method of the present invention includes the following steps: Step 1, building a deep reinforcement learning network (MA-DRL) for allocating multi-cell hybrid multi-access scenario resources, and the scenario resources include power allocation and user scheduling. Step 2, inputting the channel gain in the environment at different times into the built MA-DRL, MA-DRL calculates the channel gain difference and determines the rounds required for training based on the channel gain difference. Then the MA-DRL is trained based on the continuous interaction with the environment. Step 3, a transfer learning scheme (T-DRL) based on MA-DRL is proposed to enhance the ability of MA-DRL to cope with environmental changes. In the face of the allocation network in MA-DRL, it is divided into a power allocation subnetwork and a user scheduling subnetwork, and the two subnetworks are migrated separately. Step 4, T-DRL is divided into three migration modes, namely, a migration mode of migrating only the power allocation subnetwork, a migration mode of migrating only the user scheduling subnetwork, and a migration mode of migrating both subnetworks. Freely select the appropriate migration mode according to different environmental changes.

[0026] Specifically, the resource allocation method of the present invention includes the following steps:

[0027] Step one, reinforcement learning is applied to a multi-cell hybrid multiple access scenario, wherein the present invention uses a deep Q network (DQN) for user scheduling and a multi-cell deep policy gradient network (MADDPG) for power allocation.

[0028] The cell set in the hybrid multiple access scenario is represented by the set {1, ..., n, ..., N}, with a total of N. The user set in each cell is represented by {1, ..., m, ..., M}, with a total of M. The subchannel set is represented by {1, ..., k, ..., K}, with a total of K. In the current time slot, the signal-to-noise ratio of user m in cell n on subchannel k is expressed as

[0029]

[0030] in, represents the channel gain from user m in cell n to the base station on subchannel k in the current time slot, p n，m，t (t) is the signal power it transmits, y n，m，k (t) is the interference from other users, is the additive Gaussian white noise of this time slot. Due to the SIC principle of the NOMA system, only the signals generated by users with smaller channel gain than user m are considered as interference. Then, according to the Shannon formula, the corresponding signal rate can be expressed as

[0031] R n，m，k (t) = Blog2(1+Φ n，m，k(t)). (2)

[0032] When user m in cell n is connected to subchannel k, if user m is not interfered by other users in cell n on subchannel k, user m is an uplink user using orthogonal multiple access (OMA), otherwise user m is an uplink user using non-orthogonal multiple access (NOMA). In the uplink multi-cell hybrid multiple access scenario, the number of users connected to the same subchannel k in the same cell cannot exceed the maximum number of connections E. max ,Right now e n，m，k (t) indicates whether subchannel k is connected to user m in cell n.

[0033] First, users in all cells simultaneously start to obtain the current state information st, that is, the channel gain, and the module for calculating the interaction round is based on Start calculating the channel gain difference between the current moment and the previous moment Where H(t) is the channel gain of the user in the current time slot, is the channel gain of the user in the previous time slot. The round C required for interaction under the current channel gain is determined based on the difference, and the calculation formula is:

[0034]

[0035] Where L is the maximum number of interaction rounds, is the theoretical maximum channel gain difference. If Less than the minimum limit d min , then only one round of interaction is performed under this environment variable. This adaptive module combines online learning and offline learning to make the model better fit the environment.

[0036] Step 2: After calculating the number of interactions with the environment under the fixed current environment state, each cell is regarded as an intelligent agent in each round of interaction, and the resource allocation network of the intelligent agent is based on s t Calculate the current optimal action a t The variables that need to be defined in the hybrid multiple access are: t To represent the state of the user in the current time slot t, in this scenario, it represents the channel gain of each user uplinking to the base station of other cells. t represents the user's action at the current time slot t, Results scheduled by user And the power allocation results Composition. t Represents the reward obtained by the user in the current time slot t, which represents the rate value obtained by the user in this scenario.

[0037] The resource allocation network is based on the selected action at Interact with the environment and get the current reward r t and the next moment state information s t+1 , and then the memory block at the current moment is represented as {s t , a t , r t ,s t+1}.

[0038] Step 3: After entering the MA-DRL training process, in order to improve the training effect of MA-DRL, the value function is used to calculate the value of the memory block for the training process. The module for calculating the value of the memory block calculates the value v of the memory block at the current moment t , and compare it with the memory block {s t , a t , r t ,s t+1} are stored in the memory. t The calculation expression is

[0039]

[0040] v t C in ξ refers to the interaction round under the same environment variables, and λ refers to the sequence number of this memory block in C. t It refers to the time difference error (TD error). The method of the present invention mainly uses the TD error generated during reinforcement learning training to measure the importance of memory blocks to the training process. The resource allocation network is based on v t A certain number of memory blocks are selected from the memory bank, and the user scheduling network minimizes the loss function To update its own network parameters, The estimated reward for this slot combined with the long-term reward calculated by the representative network, where γ is the discount factor, is the optimal reward for the next time slot calculated by the network. i , a i , μ) is the actual reward of the current time slot, μ represents the parameters of the online network in the user scheduling network, and μ′ represents the parameters of the target network. The power allocation network is based on minimizing the loss function to update its own parameters. D is the number of selected memory blocks, represents the network’s estimated reward combined with the long-term reward, Q(s t+1 , a t+1 ,θ Q′ ) is the reward of the optimal action in the next time slot, Q(s i , a i ,θ Q ) is the real reward of the current time slot. In the power allocation network, θ Arepresents the parameters of the online policy network, θ A′ represents the parameters of the target policy network, θ Q represents the parameters of the online Q network, θ Q′ Represents the parameters of the target Q network.

[0041] Step 4: In a complex communication environment, the network needs to establish a certain ability to resist environmental changes. The present invention uses transfer learning to enhance the robustness of the MA-DRL model in the target domain. The present invention uses the network parameters and the entire network in the source domain as the knowledge in the source domain, and applies the knowledge to the update of the new network parameters. Different from the general reinforcement learning parameter update method, in DQN, the present invention uses and Update parameters, μ new represents the parameters of the new user scheduling network in the target domain, μ old Represents the parameters of the old user scheduling network of the source domain. Represents the online network in the new network, Represents the target network in the new network. β is the combination rate of the new and old networks. β is set small at the beginning and increases as the training process progresses.

[0042] As for the other part of T-DRL, due to the complexity and particularity of MADDPG, the present invention first migrates the MADDPG parameters of the old network. and Update the parameters of its own policy network and Q network. This is a soft update method. Every once in a while, the user scheduling network assigns the combination of the new and old policy network parameters to the Q network parameters in a certain ratio. τ represents this ratio. σ represents the combination ratio of the new and old networks. σ is set very small at the beginning and increases continuously as the migration process proceeds.

[0043] In addition to updating parameters, the user scheduling network also migrates the entire target Q network in the source domain and uses it to estimate the Q value. Q(s t+1 , a t+1 ,θ Q′ ) becomes ∈Q new (s t+1 , a t+l ,θ Q′ )+(1-∈)Q old (s t+1 , a t+1 ,θ Q' ). After updating the parameters of the entire resource allocation network, use σ = γ σ σ, β = γ β β and ∈ = γ ∈∈ updates the values ​​of β, and σ. Repeat the above process until the network converges.

[0044] T-DRL is divided into three migration modes, namely, the migration mode of migrating only the power allocation subnetwork, the migration mode of migrating only the user scheduling subnetwork, and the migration mode of migrating both subnetworks. The first migration mode only migrates the DQN network. This migration mode can be selected when the power resources of the environment do not change much. The second migration mode only migrates the MADDPG network. This migration mode can be selected when the sub-channel resources of the environment do not change much. The third migration mode migrates the DQN network and the MADDPG network at the same time. When the overall environment changes, such as the channel fading mode of the environment changes, it will affect the resource allocation of the two networks at the same time. At this time, this migration mode can be selected. The T-DRL used in the method of the present invention can freely select a suitable migration mode according to different environmental changes.

[0045] The present invention selects the total rate of all users as the optimization target, so the joint problem of user scheduling and power allocation can be expressed as

[0046]

[0047] s·t·C1:

[0048] C2:

[0049] C3:

[0050] C4

[0051] In (5), constraint C1 means that the power transmitted by each user cannot exceed the maximum power limit P max And it must be greater than 0. Constraint C2 means that in the same cell, the number of users connected to the same subchannel k cannot exceed the maximum number of connections E max .e n，m，k (t) indicates whether subchannel k is connected to user m in cell n. Constraint C3 indicates that e n，m，k (t) is between 0 and 1, e n，m，k (t) = 1 indicates its connection, e n，m，k (t) = 0 means it is not connected. Constraint C4 indicates that the rate of each user cannot be lower than the minimum rate requirement R min .

[0052] In summary, the present invention studies the problem of dynamic resource allocation with the goal of maximizing the total rate in a hybrid multiple access scenario. The present invention proposes a flexible solution to combine the hybrid multiple access scenario and solve the problem of being difficult to model in the scenario. The solution proposed by the present invention combines transfer learning and reinforcement learning, making the model of the present invention more robust in the hybrid multiple access scenario. The present invention also introduces a mechanism that combines online learning and offline learning to maintain the stability of the solution when the environmental variables change significantly. At the same time, the present invention provides a flexible transfer method to cope with environmental changes.

[0053] The performance of the method simply including reinforcement learning in the present invention is as shown in the attached Figure 3 As shown, as a comparison, the present invention proposes a centralized learning (CL) scheme and a distributed learning (DL) scheme to solve the same problem. The CL scheme has only one central agent with computing power, while the DL scheme uses completely independent agents. The present invention changes the learning rate of the power distribution network to comprehensively evaluate the performance of the method of the present invention. Since the action space of the CL scheme is too large and all actions cannot be traversed, the CL scheme cannot select the optimal action. On the contrary, the DL scheme has completely independent agents, but the agents cannot obtain the entire channel information and action information, which also affects the final performance. The agents in the present invention share their action information with each other, which enables the method of the present invention to obtain global information.

[0054] Attached Figure 4 , attached Figure 5 and attached Figure 6 The performance of the final combined with transfer learning in the method of the present invention is shown. Figure 4 The migration effect of the method of the present invention under the change of the channel fading mode is shown in FIG. Figure 5 The migration effect of the method of the present invention under the change of the initial position of the representative user. Figure 6 The migration effect of the method of the present invention under the change of the update range of the representative user location. From the above three figures, it can be seen that the three migration methods in the present invention converge faster than the method of directly placing them in the target domain without migration. In the early stage of training, the three migration methods all achieve a higher total rate. Some methods even have final performance that exceeds the method without migration. Since the knowledge of the source domain is added in the early stage of training, the method of the present invention can achieve convergence faster. At the same time, since the method of the present invention allows the old network and the new network to be updated together in the new environment after migration, it may eventually achieve a higher total rate than without migration. In addition, the present invention proposes a selective and flexible migration method, and the appropriate migration method can be selected according to the way the environment changes.

[0055] The resource allocation method proposed in the present invention aims to maximize the total rate of all users, which can effectively improve the total rate of users and achieve a faster convergence speed when combined with transfer learning.

[0056] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A resource allocation method based on transfer reinforcement learning, which is applicable to a hybrid multiple access scenario of uplink multi-cells, and is characterized by: The resource allocation method comprises the following steps: Step 1: Build a deep reinforcement learning network (MA-DRL) for resource allocation, input the channel gain of the current environment into the deep reinforcement learning network, and the deep reinforcement learning network calculates the channel gain difference and determines the round of interaction between the network and the environment based on the channel gain difference; Step 2: MA-DRL interacts with the environment. In each round of interaction, all cells obtain the current environmental status. MA-DRL, which is used to allocate power allocation and user scheduling resources, obtains resource allocation actions based on the status. MA-DRL then interacts with the environment based on the obtained resource allocation actions to obtain the current reward and the next environmental status. Step 3: Perform the training process of MA-DRL. The training will be carried out based on the experience blocks obtained from the interaction in step 2. Each experience block obtained from the interaction with the environment is stored in the memory bank and a value label is added. When extracting the memory block, the experience block with a larger value label is selected. Step 4: Based on the resource allocation network, a transfer learning scheme based on MA-DRL is proposed. The two sub-networks in MA-DRL are migrated separately. Among them, there are three types of transfer learning schemes: the first is to migrate only the power allocation sub-network, the second is to migrate only the user scheduling sub-network, and the third is to migrate both sub-networks; Step 4 is as follows: Step 4-1: Use transfer learning to enhance the robustness of the MA-DRL model in the target domain. Specifically, use the network parameters and the entire network in the source domain as the knowledge in the source domain, and apply the knowledge to the update of the new network parameters. In DQN, use and Update parameters, μ new represents the parameters of the new user scheduling network in the target domain, μ old represents the parameters of the old user scheduling network in the source domain, Represents the online network in the new network, represents the target network in the new network, β is the combination rate of the new and old networks, β is set small at the beginning and increases as the training process progresses; Step 4-2: Migrating the parameters of the old network's multi-cell deep policy gradient network (MADDPG), the present invention uses and Update the parameters of its own policy network and Q network. This is a soft update method. Every once in a while, the user scheduling network assigns the combination of the new and old policy network parameters to the Q network parameters in a certain ratio. τ represents this ratio, and σ represents the combination ratio of the new and old networks. σ is set very small at the beginning and increases continuously as the migration process proceeds. Step 4-3: The user schedules the network to migrate the entire target Q network in the source domain and uses it to estimate the Q value, specifically Q(s t+1 , a t+1 ,θ Q′ ) becomes ∈Q new (s t+1 , a t+1 ,θ Q′ )+(1-∈)Q old (s t+1 , a t+1 ,θ Q′ ), after updating the parameters of the entire resource allocation network, use σ = γ σ σ, β = γ β β and ∈ = γ ∈ ∈ updates the values ​​of β, ∈ and σ, and repeats the above process until the network converges.

2. A resource allocation method based on transfer reinforcement learning according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3-1: After entering the training process of the deep reinforcement learning network, use the value function to calculate the value of the memory block for the training process. The module that calculates the value of the memory block at the current moment calculates the value v of the memory block. t , and compare it with the memory block {s t , a t , r t ,s t+1 } are stored in the memory, where the value v t The calculation expression is C is the interaction round under the same environment variables, λ is the sequence number of this memory block in C, ξ t is the time difference error, Step 3-2: Resource allocation network according to v t A certain number of memory blocks are selected from the memory bank, and the user schedules the network by minimizing the loss function To update its own network parameters, represents the estimated reward for this slot combined with the long-term reward calculated by the network, γ is the discount factor, is the optimal reward for the next time slot calculated by the network, Q(s i , a i , μ) is the actual reward of the current time slot, μ represents the parameters of the online network in the user scheduling network, and μ′ represents the parameters of the target network; Step 3-3: Power distribution network based on minimizing loss function To update its own parameters, D is the number of selected memory blocks, represents the network’s estimated reward combined with the long-term reward, Q(s t+1 , a t+1 ,θ Q′ ) is the reward of the optimal action in the next time slot, Q(s i , a i ,θ Q ) is the real reward of the current time slot; in the power allocation network, θ A represents the parameters of the online policy network, θ A′ represents the parameters of the target policy network, θ Q represents the parameters of the online Q network, θ Q′ Represents the parameters of the target Q network.

3. The resource allocation method based on transfer reinforcement learning according to claim 1, characterized in that: The step 1 specifically includes the following steps: Step 1-1: Apply reinforcement learning to the multi-cell hybrid multiple access scenario, use a deep Q network for user scheduling, and use a multi-cell deep policy gradient network for power allocation; Step 1-2: Users in all cells start to obtain current status information s at the same time t , that is, the channel gain, the module that calculates the interaction round is based on Start calculating the channel gain difference between the current moment and the previous moment Where H(t) is the channel gain of the user in the current time slot, is the channel gain of the user in the previous time slot. The round C required for interaction under the current channel gain is determined based on the difference. The calculation formula is: Among them, the largest number of interaction rounds is is the theoretical maximum channel gain difference, if Less than the minimum limit d min , then only one round of interaction will be performed under this environment variable.

4. The resource allocation method based on transfer reinforcement learning according to claim 1, characterized in that: The specific method of step 2 is as follows: Step 2-1: After calculating the number of interactions with the environment under the fixed current environment state, each cell is regarded as an intelligent agent in each round of interaction, and the resource allocation network of the intelligent agent is based on s t Calculate the current optimal action a t , the variables that need to be defined in the hybrid multiple access are: t To represent the state of the user in the current time slot t, in this scenario, it represents the channel gain of each user uplinking to the base station of other cells, a t represents the user's action at the current time slot t, Results scheduled by user And the power allocation results Composition, r t Represents the reward obtained by the user in the current time slot t, which represents the rate value obtained by the user in this scenario; Step 2-2: Resource allocation network based on selected action a t Interact with the environment and get the current reward r t and the next moment state information s t+1 , and then the memory block at the current moment is represented as {s t , a t , r t ,s t+1 }.

Images