EMBB and URLLC service resource allocation method and system based on deep reinforcement learning

Abstract

The invention discloses an eMBB and URLLC service resource allocation method and system based on deep reinforcement learning. The method comprises the following steps: step 1, establishing a Markov decision process of eMBB and URLLC resource allocation; 2, initializing parameters; step 3, reinforcement learning training is carried out; 4, updating the evaluation neural network and the target neural network; and step 5, outputting an eMBB / URLLC service resource allocation result obtained through target neural network optimization. According to the technical scheme, the multiplexing problem of the eMBB and the URLLC in the 5G network can be better solved, the strict time delay and high reliability requirements of the URLLC service are met, meanwhile, the data transmission rate of the eMBB service is maximized, and therefore efficient transmission of the service and reasonable allocation of resources are achieved.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology, and in particular to a method and system for allocating eMBB and URLLC service resources based on deep reinforcement learning in a 5G network. Background Art

[0002] ITU has defined three major business scenarios for 5G, namely: enhanced mobile broadband (eMBB), ultra-reliable and low-latency (URLLC), and massive machine-type communications (mMTC). mMTC supports the connection of massive IoT devices, meeting the needs of scenarios such as smart homes, smart cities, and industrial automation. eMBB focuses on providing high-speed, large-capacity wireless broadband services, suitable for high-traffic applications such as high-definition video, virtual reality, and cloud games, meeting users' needs for faster and more stable networks. URLLC focuses on providing extremely low latency and high-reliability communication services, suitable for applications such as unmanned driving, remote surgery, and industrial control that have extremely high requirements for latency and reliability, ensuring the real-time and stability of mission-critical applications. In 5G network deployment, the coexistence of eMBB and URLLC is a typical application scenario. Since there is competition for spectrum resources when these two services coexist, reasonable resource allocation enabling technologies need to be adopted.

[0003] In the prior art, for resource allocation in the scenario where eMBB and URLLC coexist, a resource punching method is often used, that is, the resources of the eMBB service are punched when the URLLC service arrives, and the punched resources are used to transmit the URLLC service. Regarding the resource allocation method when eMBB and URLLC coexist, early studies mainly used optimization theory to construct a joint resource allocation optimization model. In recent years, the industry has increasingly adopted the construction of neural network models and used deep learning algorithms to solve them. At present, how to more effectively dynamically allocate appropriate resources for eMBB and URLLC services while meeting the requirements of reliability, latency and rate still requires in-depth research and has become a technical problem that needs to be solved in this field. Summary of the invention

[0004] In order to overcome the above limitations of the prior art, the present invention provides an eMBB / URLLC resource allocation method and system based on deep reinforcement learning. The technical solution of the present invention first allocates resources for eMBB services, and then punches the resources for eMBB services after the URLLC services arrive. Through the technical solution provided by the present invention, the multiplexing problem of URLLC and eMBB on the 5G network can be better solved, and the reasonable allocation of resources and efficient transmission of services can be achieved.

[0005] The present invention adopts the following technical solution:

[0006] The eMBB and URLLC service resource allocation method based on deep reinforcement learning has the following specific steps:

[0007] Step 1. Establish a Markov decision process (MDP) for the eMBB and URLLC resource allocation problem. Complete the design of the action space, state space, and reward function. Each mini-time slot eMBB resource is allocated by the base station, and the URLLC resource is determined by the resource allocation method of the present invention. For a mini-time slot, when sending a URLLC data packet, the occupied eMBB resource is punctured.

[0008] Step 2: Initialize model parameters;

[0009] Step 3: Strengthen the learning training process;

[0010] Step 4: Update the evaluation neural network and the target neural network;

[0011] Step 5: Output the eMBB / URLLC allocation result obtained by optimizing the target neural network.

[0012] Preferably, in step 1, the Markov decision process is described as a triple consisting of a state space, an action space, and a reward;

[0013] Design of action space: Possible actions a(t) in mini-slot t are collected in set A t ={0,1,…,F}, where 0 indicates that the URLLC service packet is not transmitted, and in other cases, the action indicates the channel index of the URLLC packet transmitted in the current mini-timeslot;

[0014] Design of state space: The state of each mini-slot is given by express, The state variable representing the URLLC service of mini-timeslot t, A state variable representing the eMBB service of mini-slot t; Among them, Q t represents the length of the URLLC queue in mini-time slot t, Q t ≤Q max , Q max Indicates the maximum length of the URLLC queue; Indicates the maximum number of mini-slots that the URLLC queue can wait for, l t The number of mini-slots representing the current delay of the URLLC service in mini-slot t. The current mini-time slot immediately sends the arriving URLLC data packet; is an F-dimensional vector, representing each variable S in the F spectral channelst (f), f=1,2,…,F; S t (f) indicates whether the eMBB codeword transmitted on spectrum channel f is interrupted, S t (f) = -1 means interruption, S t (f) ≥ 0 indicates non-interruption; let p t (w) represents the number of times codeword w is punctured from the beginning of the set, and C(w) represents the maximum number of mini-slots that codeword w is allowed to puncture. Then S t (f) = max(C(w) - P t (w),-1);

[0015] Design of reward function: For codeword w, define the eMBB reward function e t (w) are as follows:

[0016]

[0017] Defining the URLLC reward function Where K0 represents the penalty threshold of the system configuration when the URLLC delay constraint is violated; therefore, at the mini-slot t, the current state is s, the current action is a, and the current reward function of the next state is s' is expressed as:

[0018]

[0019] The optimization goal is to find the optimal strategy π to maximize the cumulative reward, that is, to maximize the cumulative reward function of strategy π:

[0020]

[0021] Among them, E π It means taking mathematical expectation on strategy π;

[0022] Two neural networks with the same structure but different neural network parameters θ and θ' are introduced as the evaluation neural network and the target neural network respectively. According to formula (3), the approximate Q value of executing action a in state s and the next state transition to s' is:

[0023]

[0024] Among them, γ is the forgetting factor, which is configured as a constant greater than 0 and less than 1; in the above formula, s is s t Abbreviation of s', s t+1 a is the abbreviation of a(t), s is the abbreviation of the current state, s' is the next state, and a is the current action.

[0025] Preferably, in step 2, the parameters initialized include: the maximum number of mini-slots T, the number of channels F, the set W of all eMBB codewords transmitted in mini-slot t t , the maximum length of the URLLC queue Q max , the maximum number of mini-slots that the URLLC queue can wait for Neural network parameters θ and θ', experience buffer size G, batch training dataset size G b , the maximum number of iterations S max , initial state s.

[0026] Preferably, step 3 is as follows: initialize the environment and receive the initial state s, and set the termination flag d=0; then, obtain the action of resource allocation from the evaluation neural network, take action a in the current state s to obtain a reward r, the next state is s', and combine the reward of each mini time slot to obtain the cumulative reward, and then store (s, a, r, s') in the experience buffer; determine whether the data in all URLLC data packet queues have completed the transmission, if the data in all URLLC data packet queues have completed the transmission, set the termination flag d=1, otherwise, when the data transmission in all URLLC data packet queues is not completed, return to execute the action of obtaining resource allocation from the evaluation neural network; determine whether the maximum number of iterations has been reached, if the maximum number of iterations has not been reached, enter the next iteration, if the maximum number of iterations has been reached, end.

[0027] Preferably, step 4 is as follows: when the number of samples in the buffer reaches a threshold, start updating the evaluation neural network; randomly take out some data from the experience buffer, and train the evaluation neural network parameter θ by minimizing the loss function by the gradient descent method, and obtain the maximum number of iterations S max Then, the target neural network parameters θ' are updated by hard copy evaluation of the neural network parameters θ.

[0028] Preferably, in step 4, the loss function is defined as:

[0029] L(θ)=||Y(s,a,s')-Q(s,a;θ)|| 2 (5)

[0030] Here, ||·|| represents the Euclidean distance, Q(s,a;θ) represents the estimation of the Q function of (4) using a neural network with parameter θ.

[0031]

[0032] The present invention also discloses a deep reinforcement learning-based eMBB and URLLC service resource allocation system, which is used to execute the above method and includes the following modules:

[0033] MDP establishment module: establishes the Markov decision process for eMBB and URLLC resource allocation;

[0034] Initialization module: initialize parameters;

[0035] Reinforcement learning module: conduct reinforcement learning training;

[0036] Network update module: updates the evaluation neural network and the target neural network;

[0037] Result output module: outputs the eMBB / URLLC service resource allocation results obtained through target neural network optimization.

[0038] The technical solution provided by the present invention can better solve the multiplexing problem of eMBB and URLLC in 5G networks, meet the strict latency and high reliability requirements of URLLC services, and maximize the data transmission rate of eMBB services, thereby achieving efficient transmission of services and reasonable allocation of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the coexistence of eMBB and URLLC.

[0040] Figure 2 It is a flow chart of the eMBB / URLLC resource allocation method based on DDQN in a preferred embodiment of the present invention.

[0041] Figure 3 It is a diagram of the reinforcement learning training process.

[0042] Figure 4 It is a block diagram of an eMBB / URLLC resource allocation system based on deep reinforcement learning in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to have a clearer understanding of the technical solution of the present invention, the present invention is now described in detail in conjunction with specific embodiments.

[0044] Figure 1 Schematic diagram of the eMBB and URLLC coexistence scenario.

[0045] like Figure 1As shown in the figure, there are three eMBB users in the time-frequency two-dimensional grid, namely eMBB User1, eMBBUser2 and eMBB User3, and URLLC traffic represents URLLC services. In the figure, the smallest unit in the time domain is the mini-slot, and the smallest unit in the frequency domain is the resource block (RB). It can be seen from the figure that for eMBB User1 and eMBB User2, some mini-slots are punctured by URLLC services, while for eMBB User3, all mini-slots are not punctured by URLLC services. In the figure, Frequency is the frequency.

[0046] Let the time step of the set be t∈[1,2,…,T], where T is the maximum number of mini-slots. The number of channels is denoted by F. At the beginning of each mini-slot, the base station scheduler performs resource allocation for the eMBB service, that is, completes the scheduling of each eMBB codeword. Let W t represents the set of all eMBB codewords transmitted in mini-time slot t. After completing the eMBB service resource allocation, in each mini-time slot, u Generate a URLLC packet and put it into the URLLC queue. In each mini-timeslot, take a URLLC packet from the head of the URLCC queue each time, and decide whether to transmit the URLLC packet in the current mini-timeslot based on the eMBB and URLLC coexistence resource allocation algorithm.

[0047] First, the scheduling problem is formulated as a fully observable Markov decision process (MDP). MDP can be described as a triple consisting of state space, action space, and reward.

[0048] Action space design: Possible actions a(t) in mini-slot t are collected in set A t ={0,1,…,F}, where 0 indicates that the URLLC service packet is not transmitted. In other cases, this action indicates the channel index of the URLLC packet transmitted in the current mini-timeslot.

[0049] State space design: The state of each mini-slot is given by express, The state variable representing the URLLC service of mini-timeslot t, represents the state variable of the eMBB service in mini-time slot t. Further, Where Q t represents the length of the URLLC queue in mini-time slot t, Q t ≤Q max , where Qmax Indicates the maximum length of the URLLC queue. Indicates the maximum number of mini-slots that the URLLC queue can wait for, l t Indicates the number of mini-timeslots of the current delay of the URLLC service in mini-timeslot t. Note that if The current mini-time slot immediately sends the arriving URLLC data packet. is an F-dimensional vector, representing each variable S in the F spectral channels t (f), f=1,2,…,F. S t (f) indicates whether the eMBB codeword transmitted on spectrum channel f is interrupted or non-interrupted, S t (f) = -1 means interruption, S t (f) ≥ 0 indicates non-interruption. Let p t (w) represents the number of times codeword w is punctured from the beginning of the set, and C(w) represents the maximum number of mini-slots allowed for codeword w to be punctured. Then S t (f) = max(C(w) - P t (w),-1).

[0050] Reward function design: The principle of reward function design is to minimize the number of interrupted eMBB codewords while keeping the latency of URLLC packets below a given threshold. For codeword w, define the eMBB reward function e t (w) are as follows.

[0051]

[0052] Defining the URLLC reward function Where K0 represents the penalty threshold for violating the URLLC delay constraint configured by the system. Therefore, at mini-slot t, the current state is s, the current action is a, and the current reward function for the next state is s' can be expressed as:

[0053]

[0054] The optimization goal is to find the optimal strategy π to maximize the cumulative reward, that is, to maximize the cumulative reward function of strategy π:

[0055]

[0056] Among them, E π It means taking the mathematical expectation on the strategy π.

[0057] MDP finds the optimal strategy by solving the above dynamic programming problem, and the computational complexity is high. To reduce the complexity, the present invention adopts a deep reinforcement learning algorithm based on double deep Q network (doubledeep Q-learning, DDQN) to solve the optimization objective function. The algorithm estimates the Q function based on a neural network. In order to improve stability, experience replay and target network are introduced, including two neural networks of the same structure with different neural network parameters θ and θ', which serve as the evaluation neural network and the target neural network respectively.

[0058] In the DDQN-based eMBB / URLLC resource allocation method adopted by the present invention, DDQN performs action a in state s through formula (3), and the approximate Q value of the next state transition to s' is:

[0059]

[0060] Among them, γ is the forgetting factor, which is configured as a constant greater than 0 and less than 1. In the above formula, s is s t Abbreviation of s' t+1 , a is the abbreviation of a(t). For the sake of simplicity, s is used below to represent the current state, s' represents the next state, and a represents the current action.

[0061] The process of this embodiment is described as follows Figure 2 shown.

[0062] According to the above technology, the steps of the DDQN-based eMBB / URLLC resource allocation method adopted in the preferred embodiment of the present invention are as follows:

[0063] Step 1: Establish the MDP of the eMBB and URLLC resource allocation problem. Complete the design of the action space, state space, and reward function according to the process described above; each mini-time slot eMBB resource is allocated by the base station, and the URLLC resource is determined by the resource allocation algorithm of the present invention. For a mini-time slot, when sending a URLLC data packet, the occupied eMBB resource is punctured.

[0064] Step 2: Initialize algorithm parameters. The parameters that need to be initialized include: the maximum number of mini-slots T, the number of channels F, and the set W of all eMBB codewords transmitted in mini-slot t. t , the maximum length of the URLLC queue Q max , the maximum number of mini-slots that the URLLC queue can wait for Neural network parameters θ and θ', experience buffer size G, batch training dataset size G b , the maximum number of iterations S max , initial state s;

[0065] Step 3: Reinforcement learning training process. Figure 3 At the beginning of each iteration, the algorithm initializes the environment and receives the initial state s, and sets the termination flag d = 0. Then, the action of resource allocation is obtained from the evaluation neural network. Taking action a in the current state s can obtain a reward r. The next state is s'. Combined with the rewards of each mini-time slot, the cumulative reward is obtained, and then (s, a, r, s') is stored in the experience buffer; determine whether the data in all URLLC packet queues has been transmitted. If the data in all URLLC packet queues has been transmitted, set the termination flag d = 1 and terminate this iteration in advance. On the contrary, when the data transmission in all URLLC packet queues has not been completed, the next resource allocation action will continue to be obtained from the evaluation neural network in the current iteration, that is: return to execute the action of obtaining resource allocation from the evaluation neural network; determine whether the maximum number of iterations has been reached. If the maximum number of iterations has not been reached, enter the next iteration. If the maximum number of iterations has been reached, end this step;

[0066] Step 4: Update the evaluation neural network and target neural network. When the number of buffer samples reaches the threshold G, the update of the evaluation neural network is started. Randomly take out G samples from the experience buffer. b The gradient descent method is used to minimize the loss function to train and evaluate the neural network parameters θ, and the maximum number of iterations S is reached. max Then the target neural network parameters θ are updated by hard copying the evaluated neural network parameters θ to the target neural network parameters θ'.

[0067] In step 4, the loss function is defined as:

[0068] L(θ)=||Y(s,a,s')-Q(s,a;θ)|| 2 (5)

[0069] Here, ||·|| represents the Euclidean distance, Q(s,a;θ) represents the estimation of the Q function of (4) using a neural network with parameter θ.

[0070]

[0071] Step 5: End the algorithm and output the eMBB / URLLC allocation result obtained through the target neural network optimization.

[0072] like Figure 4 As shown, this embodiment discloses an eMBB and URLLC service resource allocation system based on deep reinforcement learning, which is used to execute the method of the above embodiment, including the following modules:

[0073] MDP establishment module: establishes the Markov decision process for eMBB and URLLC resource allocation;

[0074] Initialization module: initialize parameters;

[0075] Reinforcement learning module: conduct reinforcement learning training;

[0076] Network update module: updates the evaluation neural network and the target neural network;

[0077] Result output module: outputs the eMBB / URLLC service resource allocation results obtained through target neural network optimization.

[0078] For other contents of this embodiment, please refer to the above method embodiment.

[0079] The technical solution of the present invention can better solve the multiplexing problem of eMBB and URLLC in 5G networks, meet the strict latency and high reliability requirements of URLLC services, and maximize the data transmission rate of eMBB services, thereby achieving efficient transmission of services and reasonable allocation of resources.

[0080] The specific implementation modes of the present invention are described above to facilitate the understanding of the present invention by those skilled in the art. However, the present invention is not limited to the scope of the specific implementation modes. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and the inventions and creations conceived by the present invention are all within the scope of protection.

Claims

1. The eMBB and URLLC service resource allocation method based on deep reinforcement learning is characterized by: The specific steps are as follows: Step 1: Establish a Markov decision process for eMBB and URLLC resource allocation; Step 2: Initialize parameters; Step 3: Conduct reinforcement learning training; Step 4: Update the evaluation neural network and the target neural network; Step 5: Output the eMBB / URLLC service resource allocation result obtained through target neural network optimization.

2. The method for allocating eMBB and URLLC service resources based on deep reinforcement learning as claimed in claim 1, characterized in that: In step 1, the Markov decision process is described as a triple consisting of state space, action space, and reward; Design of action space: Possible actions a(t) in mini-slot t are collected in set A t ={0,1,…,F}, where 0 indicates that the URLLC service packet is not transmitted, and in other cases, the action indicates the channel index of the URLLC packet transmitted in the current mini-timeslot; Design of state space: The state of each mini-slot is given by express, The state variable representing the URLLC service of mini-timeslot t, A state variable representing the eMBB service of mini-slot t; Among them, Q t represents the length of the URLLC queue in mini-time slot t, Q t ≤Q max , Q max Indicates the maximum length of the URLLC queue; Indicates the maximum number of mini-slots that the URLLC queue can wait for, l t The number of mini-slots representing the current delay of the URLLC service in mini-slot t. The current mini-time slot immediately sends the arriving URLLC data packet; is an F-dimensional vector, representing each variable S in the F spectral channels t (f), f=1,2,…,F; S t (f) indicates whether the eMBB codeword transmitted on spectrum channel f is interrupted, S t (f) = -1 means interruption, S t (f) ≥ 0 indicates non-interruption; let p t (w) represents the number of times codeword w is punctured from the beginning of the set, and C(w) represents the maximum number of mini-slots that codeword w is allowed to puncture. Then S t (f) = max(C(w) - P t (w),-1); Design of reward function: For codeword w, define the eMBB reward function e t (w) are as follows: Defining the URLLC reward function Where K0 represents the penalty threshold of the system configuration when the URLLC delay constraint is violated; therefore, at the mini-slot t, the current state is s, the current action is a, and the current reward function of the next state is s' is expressed as: The optimization goal is to find the optimal strategy π to maximize the cumulative reward, that is, to maximize the cumulative reward function of strategy π: Among them, E π It means taking mathematical expectation on strategy π; Two neural networks with the same structure but different neural network parameters θ and θ' are introduced as the evaluation neural network and the target neural network respectively. According to formula (3), when action a is executed in state s, the approximate Q value of the next state transition to s' is: Among them, γ is the forgetting factor, which is configured as a constant greater than 0 and less than 1; in the above formula, s is s t Abbreviation of s', s t+1 a is the abbreviation of a(t), s is the abbreviation of the current state, s' is the next state, and a is the current action.

3. The method for allocating eMBB and URLLC service resources based on deep reinforcement learning as claimed in claim 2, characterized in that: In step 2, the parameters initialized include: the maximum number of mini-slots T, the number of channels F, the set of all eMBB codewords transmitted in mini-slot t W t , the maximum length of the URLLC queue Q max , the maximum number of mini-slots that the URLLC queue can wait for Neural network parameters θ and θ', experience buffer size G, batch training dataset size G b , the maximum number of iterations S max , initial state s.

4. The method for allocating eMBB and URLLC service resources based on deep reinforcement learning as claimed in claim 3, characterized in that: Step 3 is as follows: initialize the environment and receive the initial state s, and set the termination flag d=0; then, obtain the action of resource allocation from the evaluation neural network, take action a in the current state s to obtain reward r, the next state is s', and combine the rewards of each mini-time slot to obtain the cumulative reward, and then store (s, a, r, s') in the experience buffer; determine whether the data in all URLLC packet queues has been transmitted. If the data in all URLLC packet queues has been transmitted, set the termination flag d=1. Otherwise, when the data transmission in all URLLC packet queues has not been completed, return to execute the action of obtaining resource allocation from the evaluation neural network; determine whether the maximum number of iterations has been reached. If not, enter the next iteration. If the maximum number of iterations has been reached, end.

5. The method for allocating eMBB and URLLC service resources based on deep reinforcement learning as claimed in claim 4, characterized in that: Step 4 is as follows: when the number of buffer samples reaches the threshold, start updating the evaluation neural network; randomly take some data from the experience buffer, and train the evaluation neural network parameters θ by minimizing the loss function through the gradient descent method, and then get the maximum number of iterations S max Then, the target neural network parameters θ' are updated by hard copy evaluation of the neural network parameters θ.

6. The method for allocating eMBB and URLLC service resources based on deep reinforcement learning as claimed in claim 5, characterized in that: In step 4, the loss function is defined as: L(θ)=||Y(s,a,s')-Q(s,a;θ)|| 2 (5) Here, ||·|| represents the Euclidean distance, Q(s,a;θ) represents the estimation of the Q function of (4) using a neural network with parameter θ.

7. An eMBB and URLLC service resource allocation system based on deep reinforcement learning, used to execute the method according to any one of claims 1 to 6, characterized in that: Includes the following modules: MDP establishment module: establishes the Markov decision process for eMBB and URLLC resource allocation; Initialization module: initialize parameters; Reinforcement learning module: conduct reinforcement learning training; Network update module: updates the evaluation neural network and the target neural network; Result output module: outputs the eMBB / URLLC service resource allocation results obtained through target neural network optimization.

Images