Method for joint uplink and downlink slice resource allocation in TDD network

Abstract

A method for joint uplink and downlink slice resource allocation in a TDD network constructs a wireless access network simulation environment. Within a single step, resources are scheduled for users in each served slice. Then, in the current step, the state obtained from the previous single step is input into a neural network instantiated by a deep deterministic policy gradient algorithm to obtain an intelligent agent. The neural network then generates an action acting on the wireless access network simulation environment in the current step. After the environment runs, the updated state and the reward corresponding to the executed action are fed back. The state, action, reward, and State' are stored as an experience in the intelligent agent's memory pool. When the number of stored experiences reaches a set threshold, these experiences are extracted to train the intelligent agent. After the training is complete, the trained agent is tested online. Based on a deep deterministic policy gradient algorithm, this method avoids unexpected overflows in a particular slice that could lead to a degradation in the service level agreement (SLA).

Description

Technical Field

[0001] The present invention relates to a technology in the field of wireless resource allocation, specifically a method for joint uplink and downlink slice resource allocation in a TDD network. Background Art

[0002] Existing methods for allocating time and frequency resources in wireless access networks include static, semi-static, and dynamic allocation. Static resource allocation allocates fixed resources, which can achieve complete isolation between slices. However, the dynamic changes in user service requirements and the time-varying nature of the network environment will cause the resource demands of network slices to constantly change, resulting in low network resource utilization and inflexible resource sharing. Dynamic allocation refers to the network dynamically allocating and adjusting all available wireless resources between slices by jointly considering the dynamic changes in user service requirements, the differences between different services, and the time-varying nature of the network environment. This is relatively complex. Semi-static allocation is a compromise solution that pre-allocates a fixed portion of network resources to each slice according to certain rules, and then dynamically allocates the remaining resources to each slice through scheduling. Summary of the Invention

[0003] In response to the shortcomings of existing technologies, such as the inability to dynamically adjust the total bandwidth of V2V slices and the allocation of V2V link resources to adapt to fluctuations in business demand, and the significant limitations in adapting to changes in the number of users, the present invention proposes a method for joint uplink and downlink slice resource allocation in TDD networks. Based on the deep deterministic policy gradient (DDPG) algorithm, it can use less information as the state input of reinforcement learning. The actions output by the neural network configure the ratio of uplink and downlink resources and the bandwidth resource allocation ratio of each uplink / downlink slice. This action maintains the resource partitioning structure of multiple slices aligned in the uplink / downlink frame structure, and ensures load balancing between uplink and downlink slices as much as possible, avoiding the accidental overflow of a slice and the resulting degradation of the SLA.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a method for joint uplink and downlink slice resource allocation in a TDD network. By constructing a wireless access network simulation environment, resource scheduling is performed for users of each served slice in a single step. Then, in the current step, the state obtained by running the previous step is input into a neural network of an intelligent agent instantiated by a deep deterministic policy gradient algorithm (DDPG). The wireless access network simulation environment in which actions act on the current step is obtained through the neural network. After the environment runs, the updated state (State') and the reward corresponding to the executed action are fed back. The state (State), action (Action), reward (Reward), and State' are stored as an experience in a memory pool of the intelligent agent. When the amount of stored experience reaches a set threshold, the experience is extracted to train the intelligent agent. After the training of the intelligent agent is completed, the trained intelligent agent is tested online.

[0006] Technical Effects

[0007] This solution, based on the Deep Deterministic Policy Gradient (DDPG) algorithm, aligns the uplink and downlink frame structures of multiple slices in the time domain and then allocates bandwidth resources to different slices in the frequency domain. Compared with existing technologies, this solution achieves a new multi-slice frame structure alignment while balancing the load across multiple slices, ensuring near-optimal multi-slice SLAs. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Schematic diagram of a slice set according to an embodiment;

[0009] Figure 2 Flowchart of the present invention;

[0010] Figure 3 Schematic diagram of frequency resource allocation in the embodiment;

[0011] Figure 4 and Figure 5 Schematic diagram of the embodiment effect. DETAILED DESCRIPTION

[0012] like Figure 1 As shown, this embodiment relates to a TDD wireless access network, in which the slice set used is S = {1, 2, ..., N}, there are N slices in the network, and the transmission direction set D = {dl, ul}, where dir = dl represents downlink, and dir = ul represents uplink.

[0013] This embodiment relates to a method for joint uplink and downlink slice resource allocation in a TDD network. By constructing a wireless access network simulation environment, that is, instantiating an intelligent agent through a deep deterministic policy gradient algorithm (DDPG), the intelligent agent receives the slice service request amount, slice SLA satisfaction rate, and slice load rate from the wireless access network as state inputs, and outputs actions through a neural network to perform uplink and downlink resource allocation ratios, bandwidth resource allocation ratios for each downlink slice, and bandwidth resource allocation ratios for each uplink slice.

[0014] like Figure 2 As shown, this embodiment relates to a method for joint uplink and downlink slice resource allocation in a TDD network, specifically including:

[0015] Step 1: Build a wireless access network simulation environment and generate an intelligent agent through DDPG. In the initial state, the wireless access network simulation environment configures the corresponding environment according to a randomly generated desirable action. For the users of each slice being served, resource scheduling is performed according to the resource partitioning structure given by the action, and the corresponding state is obtained.

[0016] The agent generated by DDPG includes: Actor neural network and Critic neural network, and their network structures include an input layer, two hidden layers and an output layer, where the input layer dimension is related to the number of slices in the wireless access network, as shown in Table 1:

[0017] Table 1

[0018]

[0019]

[0020] Since the parameters collected by each slice include the service request amount M in a certain transmission direction s，dir , SLA satisfaction rate Q for each slice uplink and downlink s，dir , the uplink and downlink slice utilization rate of each slice E s，dir There are 3 parameters in total, so a single uplink and downlink slice needs to collect 6 parameters, so the input layer dimension of the Actor network is 6*N, that is, the dimension of State is 6*N

[0021] The action output by the agent includes: time domain resource division ratio λ tdd , downlink slice allocation ratio λ 1，dl ,λ 2，dl ...λ N，dl and the uplink slice allocation ratio λ 1，ul ,λ 2，ul ...λ N，ul , so the dimension of the action is 1+2N, and the input dimension of the Critic network input layer is equal to 6*N+1+2N.

[0022] The construction of the wireless access network simulation environment further includes: setting the parameters of the wireless access network including service radius, base station transmission power, air interface path loss function, user terminal transmission power, air interface bandwidth, and configuring the scheduling algorithm within each slice.

[0023] The λ tdd It is a value of 0 to 1, which determines the uplink and downlink allocation ratio of OFDM symbols contained in a subframe in the wireless time-frequency resource. The first λ of all OFDM symbols tdd Part is used for downlink transmission, and the last 1-λ of all OFDM symbols tdd Part of it is used for uplink transmission.

[0024] The parameter λ 1，dl ,λ 2，dl ...λ N，dl They represent the bandwidth ratios allocated to each downlink slice in the frequency domain, and the sum of the parameters is 1.

[0025] The parameter λ 1，ul ,λ 2，ul ...λ N，ul They represent the bandwidth ratios allocated to each uplink slice in the frequency domain, and the sum of the parameters is 1.

[0026] The users of each slice being served refer to: users in slices with different service level agreements (SLA), for example, users of eMBB slices need to implement the large-bandwidth data services agreed in the agreement, and users of URLLC slices need to implement the high-reliability and low-latency data services agreed in the agreement.

[0027] The resource division structure given by the action refers to: a method for dividing the wireless air interface time-frequency resources, according to the action Action, aligning the uplink and downlink frame structures of multiple slices in the time domain, and then allocating bandwidth resources to different slices in the frequency domain.

[0028] The scheduling algorithm within each slice refers to: when different slices schedule resources for users within the slice, they select an existing scheduling algorithm. The optional scheduling algorithms include the polling scheduling algorithm (RR) and the earliest deadline first scheduling algorithm (EDF).

[0029] Step 2: The agent interacts with the simulation environment and conducts K rounds of training (Episode). Each Episode contains M steps. A converged neural network is obtained through interactive training with the simulation environment.

[0030] The Step refers to the process in which the agent outputs an Action based on the state of the wireless access network environment, acts on the wireless access network environment, obtains corresponding rewards, and stores experience.

[0031] The said round (Episode) refers to the continuous execution of M single steps. During the process, the wireless access network environment continues to change according to the input action. Before the next round starts, the wireless access network environment will be reinitialized to the initial state.

[0032] The interaction means that in this step, the state obtained from the previous single-step operation is input into the neural network of the intelligent agent, and a new action is obtained through the neural network and applied to the current step of the wireless access network simulation environment. The environment feedback updates the state State' and the reward corresponding to the executed action, and stores the State, Action, Reward, and State' as an experience in the memory pool of the intelligent agent. When the amount of stored experience reaches the set threshold, the experience is extracted to train the intelligent agent.

[0033] The reward is the score given by the environment to the action of the agent to guide the agent to make better decisions. The calculation method is: after the round of scheduling is completed, the number of uplink and downlink service requests M of each slice in the round is counted. s，dir , SLA satisfaction rate Q for each slice uplink and downlink s，dir , the uplink and downlink slice utilization rate of each slice E s，dir As the state, the corresponding reward value R = ∑ s∈S ∑ dir∈D α s，dir ×Q s，dir -β×var(∪ s∈S ∪ dir∈D E s，dir ), where: s，dir is the QoS weight of slice s in the dir direction, Q s，dir is the QoS value of slice s in the dir direction, E s，dir is the resource utilization of slice s in the dir direction, var is the variance function, and β is the weight of the variance term. By taking QoS as the positive part of the reward, the agent is guided to ensure the service quality of the slice. By taking the load balance of the slice, that is, the variance term, as the deduction item of the reward, the agent is guided to ensure load balance between slices.

[0034] Step 3: Test the interaction between the trained agent and the simulation environment. The specific steps are as follows:

[0035] During the test phase, the uplink and downlink service request volume M of each slice in this round is collected and counted. s，dir, SLA satisfaction rate Q for each slice uplink and downlink s，dir , the uplink and downlink slice utilization rate of each slice E s，dir , the reward value R is used to evaluate the performance of the trained agent.

[0036] The proposed method was experimentally tested in a wireless communication simulation environment implemented in Python. The wireless communication environment parameters included the following: the path loss function was L(d) = 128.1 + 37.6log10(d), where d is the distance between the base station and the user in kilometers. The radio access network transmit power was 20 watts, and the user equipment (UE) transmit power was 2 watts. The radio access network contained two types of slices: eMBB and URLLC. The eMBB downlink SLA was a minimum achievable rate of 2 Mbps, and the eMBB uplink SLA was a minimum achievable rate of 0.2 Mbps. The URLLC downlink SLA was a latency of 5 milliseconds, and the URLLC uplink SLA was a latency of 5 milliseconds. The weight α for each slice was 2.5, and β was set to 10, meaning the maximum reward score was 10. eMBB used the Round Robin (RR) scheduling algorithm, while URLLC used the Earliest Deadline First (EDF) scheduling algorithm.

[0037] like Figure 3 The figure shows an example of how to partition time-frequency resources based on the actions output by the agent using reinforcement learning (DDPG) in this embodiment. This example includes two slices, eMBB and URLLC. Training is done for 100 episodes, each containing 10 steps, and a learning rate of 0.001.

[0038] Compared with the existing technology, the present invention performs hard slicing of resources and proposes a new multi-slice resource partitioning structure. Traditional algorithms are used for internal resource scheduling. However, this algorithm for joint uplink and downlink and multi-slice bandwidth allocation only requires parameters including the slice SLA satisfaction rate, slice resource utilization, and the service request volume of each slice. The DDPG algorithm used can adapt to changes in the number of users and load, and the lightweight neural network will also be conducive to deployment in base stations.

[0039] like Figure 4As shown, the present invention can guarantee the slice SLA under different loads, that is, under different loads, the exhaustive search algorithm (Exhausted-Search), the proposed DDPG, the dynamic SE-CSDR algorithm (Dynainc-SE-CSDR) with aligned uplink and downlink subcarrier allocation between slices, and the static SE-CSDR algorithm (Static-SE-CSDR). When the number of terminals reaches about 40 / 60 (eMBB / URLLC), the system has reached the critical point of full load. In order to maximize the global SLA, the present invention allocates spectrum resources to the URLLC slice, slightly sacrificing the performance of the eMBB slice, because the SLA of URLLC is a 0,1 problem, while the SLA satisfaction rate of eMBB is a gradual loss problem. The performance of this algorithm is close to optimal when it is less than full load, and the performance is much better than the static allocation algorithm when overloaded. The static method cannot be dynamically adjusted in time with changes in traffic requests, resulting in premature overflow of slices.

[0040] like Figure 5 As shown, the present invention can achieve load balancing while ensuring the slice SLA satisfaction rate. By comparing the impact of rewards with and without variance deduction items on the performance of the intelligent agent, it mainly refers to the impact of load balancing between slices, which are respectively denoted as Proposed-DDPG-with var and Proposed-DDPG-withoutvar. The horizontal axis indicates the number of different users in the experiment, and the vertical axis indicates the variance of the slice load rate to describe the load balancing situation between slices. The results show that rewards with deduction items can guide the intelligent agent to maintain load balance between slices. At the critical load point, most slices are in a high load state, so the variance is small, which is conducive to avoiding accidental overflow of a slice.

[0041] Compared with the existing technology, under the multi-slice resource partitioning structure of this method, good load balancing is maintained and the SLA satisfaction rate of the slice is guaranteed to reach 0.99 when the wireless network demand is not overloaded. Under the given reward expression, the reward is 9.9 (full score 10), which is close to the optimal.

[0042] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for joint uplink and downlink slice resource allocation in a TDD network, characterized in that: By building a wireless access network simulation environment, resources are scheduled for users of each slice being served within a single step. In the current step, the state obtained from the previous single step is input into the neural network of the agent instantiated by the Deep Deterministic Policy Gradient (DDPG) algorithm. The action is then applied to the wireless access network simulation environment of the current step through the neural network. After the environment runs, the updated state (State') and the reward corresponding to the executed action are fed back. The state (State), action (Action), reward (Reward), and State' are stored as an experience in the agent's memory pool. When the amount of stored experience reaches a set threshold, the experience is extracted to train the agent. After the training of the agent is completed, the trained agent is tested online, specifically including: Step 1: Build a wireless access network simulation environment and generate an agent using DDPG. In the initial state, the wireless access network simulation environment randomly generates a desirable action and configures the environment accordingly. It then schedules resources for each user in the served slice based on the resource partitioning structure given by the action and obtains the corresponding state. Step 2: The agent interacts with the simulation environment and conducts K rounds of training. Each episode contains M steps. A converged neural network is obtained through interactive training with the simulation environment. Step 3: Interact the trained agent with the simulation environment for testing. Specifically, during the test phase, collect and count the uplink and downlink service requests for each slice in the round. , Uplink and downlink SLA satisfaction rate of each slice , Uplink and downlink slice utilization of each slice , reward value , used to evaluate the performance of the trained agent; The round is defined as the continuous execution of M single steps, during which the wireless access network environment continuously changes according to the input action. Before the next round begins, the wireless access network environment is reinitialized to its initial state. The agent generated by DDPG includes: Actor neural network and Critic neural network, and their network structures both include an input layer, two hidden layers and an output layer; The input layer dimension of the Actor network is , that is, the dimension of State is ; The action output by the agent includes: time domain resource allocation ratio , Downlink slice allocation ratio 、 … and upstream slice allocation ratio 、 … , so the dimension of the action is , then the input dimension of the Critic network input layer is equal to ; The It is a value between 0 and 1, which determines the uplink and downlink allocation ratio of OFDM symbols contained in a subframe in the wireless time-frequency resource. Part is used for downlink transmission, and the latter part of all OFDM symbols is used for downlink transmission. Part of it is used for uplink transmission; The parameters described 、 … They represent the bandwidth ratio of each downlink slice allocated in the frequency domain, and the sum of the parameters is 1; The parameters described 、 … They represent the bandwidth ratio of each uplink slice allocated in the frequency domain, and the sum of the parameters is 1; The said construction of the wireless access network simulation environment further includes: setting the parameters of the wireless access network including service radius, base station transmit power, air interface path loss function, user terminal transmit power, air interface bandwidth, and configuring the scheduling algorithm within each slice; The resource partitioning structure provided by the action is: a method for partitioning the time-frequency resources of the wireless air interface, according to the action, aligning the uplink and downlink frame structures of multiple slices in the time domain, and then allocating bandwidth resources to different slices in the frequency domain; The scheduling algorithm within each slice refers to: when different slices schedule resources for users within the slice, they select an existing scheduling algorithm. The optional scheduling algorithms include the round-robin scheduling algorithm (RR) and the earliest deadline first scheduling algorithm (EDF); The interaction means that in this step, the state obtained from the previous single step is input into the agent's neural network, a new action is obtained through the neural network and applied to the current step of the wireless access network simulation environment. The environment feedback updates the state State' and the reward corresponding to the executed action, and the State, Action, Reward, and State' are stored as an experience in the agent's memory pool. When the amount of stored experience reaches a set threshold, the experience is extracted to train the agent. The reward is the score given by the environment to the action given by the agent to guide the agent to make better decisions. The calculation method is: after the round of scheduling is completed, the uplink and downlink service requests of each slice in the round are counted. , Uplink and downlink SLA satisfaction rate of each slice , Uplink and downlink slice utilization of each slice As the state, calculate the corresponding reward value ,in: slice exist QoS weight in the direction, slice exist QoS value in the direction, slice exist Resource utilization in the direction, var is the variance function, The weight of the variance term guides the agent to ensure the service quality of the slice by taking QoS as the positive part of the reward, and guides the agent to ensure load balance between slices by taking the load balance of the slice, that is, the variance term, as the deduction item of the reward.

Images