Method and system for resource allocation based on coordinated multipoint transmission and deep reinforcement learning, and storage medium

Abstract

The application provides a resource allocation method and system based on coordinated multipoint transmission and deep reinforcement learning and a storage medium, the resource allocation method comprising: step 1: introducing a BLER interruption probability for URLLC users, modeling an eMBB and URLLC joint resource allocation problem as a CMDP problem; step 2: adopting a twin delayed deep deterministic policy gradient algorithm based on constraint correction strategy optimization, training offline, so that the algorithm can utilize forward propagation online to find an optimal resource allocation strategy with low complexity and reduce the processing delay of the algorithm. The application has the beneficial effects that the URLLC delay and reliability requirements in a multi-cell network can be met, and the QoS indicators of URLLC in a high mobility scenario can be met. At the same time, by dynamically regulating the URLLC pilot length and the number of data transmission symbols, the attenuation of the eMBB user performance caused by the resource occupation of the URLLC user can be effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication technology, and particularly relates to a resource allocation method and system based on coordinated multipoint transmission and deep reinforcement learning, and a storage medium. BACKGROUND

[0002] eMBB and URLLC are two typical scenarios in the fifth-generation (5G) and Beyond 5G (B5G) mobile communications. Among them, eMBB mainly faces the mobile Internet with explosive growth of traffic, providing high-bandwidth services with peak rate of gigabit level, such as 8K video services, augmented reality services, etc. URLLC mainly faces mission-critical applications in vertical industries, such as remote medical treatment, unmanned driving, smart power grid, etc. They have very strict requirements on reliability and latency. Now, URLLC requires 99.99% reliability and 5ms end-to-end (E2E) latency for transmitting 32-byte data packets from 5G, and evolves to 99.9999% reliability and 1ms end-to-end latency in B5G. At the same time, there are also a large number of multi-service fusion applications (such as mixed reality) in B5G, which require both low-latency high-reliability URLLC services and high-bandwidth eMBB services. In order to effectively solve the coexistence problem of these two types of services and meet their respective performance requirements, URLLC needs to preempt eMBB transmission resources (also known as "punching"). This mechanism aims to sacrifice the data rate of eMBB users to meet the stringent reliability and latency requirements of URLLC, and to realize the coexistence of the two types of services.

[0003] On the other hand, Constrained Markov decision process (CMDP) is very effective in handling resource scheduling in dynamic environments, so this method is commonly used to model resource allocation problems in eMBB and URLLC coexistence scenarios. However, in actual scenarios, the CMDP problem often contains a state space and an action space with huge dimensions, making the complexity of the solution algorithm extremely high. The problem brought about is that when the time for the algorithm to obtain the resource scheduling strategy exceeds the coherence time of the channel, the strategy will expire and is no longer applicable, so the practicality is poor.

[0004] For the resource allocation of URLLC and eMBB coexistence scenarios, the existing scheme is mainly based on single-cell network, and the more practical multi-cell network is rarely considered. We also notice that for 5G / B5G multi-cell network, CoMP technology has received more and more attention. CoMP technology uses multiple base stations to send the same information to the same user (often a cell edge user) to reduce the interference received by the edge user, thereby improving its throughput performance. However, the existing multi-cell network resource allocation scheme based on CoMP technology only considers single-service scenarios, i.e., eMBB or URLLC, and does not consider the joint resource allocation of URLLC and eMBB. At the same time, due to the randomness of the wireless channel and the dynamics (such as mobility) of the URLLC user, it is difficult to guarantee the Quality of Service (QoS) requirements of URLLC with probability 1 in practice. The existing URLLC resource puncturing scheme is more to meet the block error rate of each URLLC packet with probability 1, i.e., the instantaneous reliability constraint of URLLC, which will lead to the dilemma that eMBB resources are heavily punctured in extreme cases, while URLLC reliability cannot be met, resulting in waste of eMBB resources. SUMMARY

[0005] In order to solve the problems in the prior art, the present application provides a resource allocation method based on coordinated multi-point transmission and deep reinforcement learning, comprising the following steps:

[0006] Step 1: introduce the BLER interruption probability for URLLC users, and model the joint resource allocation problem of eMBB and URLLC as a CMDP problem;

[0007] Step 2: adopt a twin-delayed deep deterministic policy gradient algorithm based on constraint correction strategy optimization, which can find the optimal resource allocation strategy by using deep neural network forward propagation with low complexity online, thereby reducing the processing delay of the algorithm.

[0008] As a further improvement of the present application, in step 1, the following steps are included:

[0009] Step S1: based on the imperfect channel mutual independence, the imperfectness of channel estimation delay, and the imperfectness of channel estimation error, the mathematical modeling of the imperfect channel is obtained as follows:

[0010]

[0011] wherein

[0012] Step S2: find the signal-to-interference-and-noise ratio of the URLLC user under the imperfect channel, which is expressed as follows:

[0013]

[0014] where represents the impact of imperfect channel, is the precoding vector, κ[t] is the interference signal, n[t] is the Gaussian white noise;

[0015] Step S3: assuming that the short packet length of the URLLC data packet is Z, when the RRH cluster encodes the URLLC packet into l x OFDM symbols at time t, the achievable block error rate is approximated by the short packet formula as follows:

[0016]

[0017] where W0, T0 are the subcarrier spacing and the single symbol duration respectively, R[t] = Z / (l x [t]W0T0) is the actual transmission rate of the URLLC packet, is the channel dispersion;

[0018] Step S4: define the BLER outage probability, and model the CMDP problem.

[0019] As a further improvement of the application, in the step S4, further comprising:

[0020] Step 1: give the BLER violation indicator function c[t] at time t, defined as At the same time, maximize the number of eMBB symbols, and the reward function is defined as:

[0021] ξ[t] = L max -l p [t]-l x [t] (1.7);

[0022] Step 2: under the deterministic policy μ, the long-term reward of eMBB is obtained as follows:

[0023]

[0024] The long-term reliability constraint of URLLC is expressed as follows:

[0025]

[0026] where Γ ∈ (0, 1] is the discount factor.

[0027] As a further improvement of the application, in the step 1, if the BLER is higher than the given threshold ε th , the indicator function c[t] = 1, otherwise the indicator function c[t] = 0.

[0028] As a further improvement of the present application, in the step 2, further comprising:

[0029] Step one: initialize the parameters of two groups of neural networks, including action network group and evaluation network group, wherein the action network group is responsible for learning and feeding back URLLC pilot length and data symbol number, denoted as action vector a[t] = [l p [t],l x [t]] and the evaluation network group is responsible for evaluating the action vector;

[0030] Step two: according to the current real uplink channel gain Obtain a set of observation values According to the observation value, a sample transition space composed of the current state, the corresponding action, the corresponding reward and cost, and the next state is obtained, denoted as <o[t],a[t],ξ[t],c[t],o[t+1]> and stored in the experience replay memory;

[0031] Step three: repeat the step two until the experience replay memory is filled, and then execute step four;

[0032] Step four: randomly sample N s group transition sample data in the buffer, calculate the time difference error, and update the evaluation group network;

[0033] Step five: use the evaluation group network to calculate the URLLC long-term reliability constraint, if the constraint is violated, the action group network uses the stochastic gradient descent to minimize the long-term reliability constraint, otherwise uses the stochastic gradient ascent to maximize the long-term reward.

[0034] As a further improvement of the present application, in the step S1, the imperfect channel reciprocity is:

[0035] In the TDD system, the channel vectors of the uplink and downlink are modeled as follows:

[0036]

[0037] Wherein h j [t] and h [t] respectively represent the channel vectors of the uplink and downlink at time t, v [t] describes the uncertainty of imperfect channel reciprocity, and φ ∈ [0, 1] represents the channel reciprocity coefficient;

[0038] Imperfectness of channel estimation delay:

[0039] In the CoMP system, the formula for expressing the influence of channel estimation delay in mathematical language is as follows:

[0040]

[0041] Wherein is the real channel at time (t-τ), τ represents the channel estimation delay, is a random vector, b is the channel correlation coefficient;

[0042] Imperfectness of channel estimation error:

[0043] The channel estimation error is expressed as the difference between the real channel and the estimated channel, expressed as Here represents the estimated channel at time (t-τ), and the channel estimation error e j,τ is a random vector, which obeys a circularly symmetric complex Gaussian distribution, that is The variance The mathematical form is as follows:

[0044]

[0045] Here ρ j [t]=α j [t]β j , α j [t] and β j represent the path loss and the maximum transmission power of the RRH at time slot t, respectively, represents the noise variance, l p [t] indicates the pilot length allocated to the URLLC user at time slot t.

[0046] The application also discloses a resource allocation system based on coordinated multipoint transmission and deep reinforcement learning, comprising a memory, a processor and a computer program stored in the memory, the computer program being configured to realize the steps of the resource allocation method of the application when called by the processor.

[0047] The application also discloses a computer readable storage medium, which stores a computer program configured to realize the steps of the resource allocation method of the application when called by a processor.

[0048] The application has the following advantages: 1. The application can not only meet the URLLC delay and reliability requirements in a multi-cell network, but also meet the QoS indicators of URLLC in a high mobility scenario. At the same time, by dynamically regulating the URLLC pilot length and the number of data transmission symbols, the application can effectively reduce the degradation of eMBB user performance caused by URLLC preemption; 2. The channel estimation delay of the application is within 2ms, and the application can still effectively meet the 1ms delay and 99.9999% reliability requirements of URLLC users compared with existing schemes, while obtaining good eMBB performance. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a CoMP network diagram with two RRH clusters of the application;

[0050] Figure 2 is a packet transmission mechanism of the coordinated multi-point transmission system of the application;

[0051] Figure 3 is a twin-delay deep deterministic policy gradient algorithm based on CRPO of the application; DETAILED DESCRIPTION

[0052] Noun explanation:

[0053] URLLC: Ultra-reliable and low-latency communications, ultra-reliable and low-latency;

[0054] eMBB: Enhanced mobile broadband, enhanced mobile broadband;

[0055] CoMP: Coordinated multipoint, coordinated multi-point transmission;

[0056] DRL: Deep reinforcement learning, deep reinforcement learning;

[0057] RPS-U: Resource puncturing scheme for URLLC, URLLC resource puncturing scheme;

[0058] CRPO: Constraint-rectified policy optimization, constraint-rectified policy optimization;

[0059] BBU: Baseband unit, baseband unit;

[0060] RRH: Remote radio head, remote radio head;

[0061] CMDP: Constrained Markov decision process, constrained Markov decision process;

[0062] The application faces the coexistence scene of multi-cell, high mobility, ultra-reliable and low-latency (URLLC) and enhanced mobile broadband (eMBB) services, and proposes a resource allocation scheme based on coordinated multipoint (CoMP) and deep reinforcement learning (DRL), also known as resource puncturing scheme for URLLC (RPS-U). By considering channel imperfect mutual independence, estimation error, estimation delay and interval interference, the application can dynamically control the number of URLLC pilot symbols and data symbols, not only realizing the ultra-low latency and ultra-high reliability requirements of URLLC, but also effectively reducing the resources occupied by eMBB, and improving the block error rate interruption performance of URLLC. The application is mainly applied to resource multiplexing of ultra-reliable and low-latency communication and enhanced mobile broadband services in 5G / B5G.

[0063] The resource allocation method based on coordinated multipoint transmission and deep reinforcement learning disclosed in the application comprises the following steps:

[0064] Step 1: Considering the influence of wireless channel deep fading and user dynamics, the BLER interruption probability for URLLC users is introduced, and the joint resource allocation problem of eMBB and URLLC is modeled as a CMDP problem; compared with the traditional scheme focusing on short-term benefits and instantaneous constraints, the scheme focuses more on long-term eMBB throughput and aims to guarantee the long-term reliability constraint of URLLC, allowing the reliability of URLLC users to be unable to meet in extreme cases, thereby improving the long-term throughput performance of eMBB.

[0065] Step 2: Considering the real-time of resource scheduling strategy, a CRPO-based TD3 algorithm is adopted, which is a twin-delayed deep deterministic policy gradient algorithm (CRPO-based TD3 algorithm) optimized by constraint correction strategy, through offline training, the algorithm can find the optimal resource allocation strategy with low complexity by using deep neural network (DNN) forward propagation online, thereby reducing the processing delay of the algorithm.

[0066] Specifically introduced:

[0067] The present application considers a two-tier multi-cell CoMP network, where the first tier component is the baseband unit (BBU) and the second tier component is the remote radio head (RRH), which serves both URLLC and eMBB users. The present application considers that each two RRHs form a RRH cluster to serve URLLC users, and each RRH is equipped with N x antennas, and each user is equipped with one antenna. All RRHs are connected to the BBU through fiber, as shown in Figure 1 .

[0068] The time-frequency resource of each slot is divided into L max orthogonal frequency-division multiplexing (OFDM) symbols. Generally, the eMBB user resource is scheduled once at the beginning of each slot, and the URLLC user resource is punctured to the eMBB user at the beginning of each mini-slot. In the same RRH cluster of the CoMP system, two cooperative RRHs use the same time-frequency resource to send the same data packet to the users they serve, as shown in Figure 2 .

[0069] In order to achieve the URLLC latency and reliability requirements in complex scenarios (inter-cell interference, user dynamics, real-time strategy), and to protect the eMBB user resource, the present application considers imperfect channels and inter-cell interference, and models the joint resource allocation problem of eMBB and URLLC as a CMDP problem, which aims to improve the ability of the present application to resist non-ideal environments. At the same time, the present application combines deep reinforcement learning algorithm with CRPO algorithm, so that the present application can solve the CMDP problem and has the ability to quickly feedback to the environment.

[0070] Due to the wide application of massive multiple-input and multiple-output (mMIMO) technology in future 5G / B5G, time division duplex (TDD) communication system has become the first choice in 5G / B5G, because TDD system can significantly reduce the channel training overhead of multi-antenna system. Based on this situation, the present application considers imperfect channels based on TDD system, and finally characterizes the URLLC signal-to-interference-plus-noise ratio (SINR) and BLER outage probability.

[0071] In high mobility scenarios, the perfect channel reciprocity assumed in TDD does not hold anymore. The channel vectors of the uplink and downlink can be modeled as follows:

[0072]

[0073] where and denote the channel vectors of the uplink and downlink at time t, respectively, and j v [t] depicts the uncertainty of imperfect channel reciprocity, and φ ∈ [0, 1] represents the channel reciprocity coefficient.

[0074] In a real CoMP system, in order to realize the coordinated communication among RRHs, all RRHs in each RRH cluster will share the channel information through the front-haul link after the channel estimation is completed, thus there is a channel estimation delay, denoted as τ. The influence of the channel estimation delay is expressed in mathematical language as follows:

[0075]

[0076] Here is the real channel at time (t-τ), is a random vector, and b is the channel correlation coefficient. The channel correlation coefficient is mainly related to the maximum Doppler shift and the channel estimation delay, and can be modeled as the zeroth-order first kind Bessel function.

[0077] In addition to the imperfect channel reciprocity and the channel estimation delay, the channel estimation error also causes channel imperfection. The channel estimation error is expressed as the difference between the real channel and the estimated channel, expressed as Here denotes the estimated channel at time (t-τ). At the same time, the channel estimation error e j,τ is a random vector, which is subject to a circularly symmetric complex Gaussian distribution, i.e. with variance The mathematical form of the channel estimation error is as follows:

[0078]

[0079] Here ρ j [t] = α j [t]β j , α j [t] and β j represent the path loss and the maximum transmission power of the RRH at time slot t, respectively, denotes the noise variance, and l p [t] represents the pilot length allocated to the URLLC user at time slot t.

[0080] By comprehensively considering the above three kinds of channel imperfections, the mathematical modeling of the imperfect channel can be obtained as follows:

[0081]

[0082] where

[0083] The precoding considers maximum ratio transmission (MRT), thus the signal-to-interference-plus-noise ratio of URLLC users under imperfect channel is expressed as follows:

[0084]

[0085] where represents the impact of imperfect channel, is the precoding vector, κ[t] is the interference signal, and n[t] is the Gaussian white noise.

[0086] The present application considers that URLLC data packets are usually short packets, assuming the packet length as Z, when the RRH cluster compiles the URLLC packet into l x [t] OFDM symbols at time t, the achievable block error rate can be approximated by the short packet formula as

[0087]

[0088] where W0, T0 are the subcarrier spacing and single symbol duration, respectively, and R[t] = Z / (l x [t]W0T0) is the actual transmission rate of the URLLC packet, is the channel dispersion.

[0089] According to the above expression, the BLER outage probability is defined next, first giving the BLER violation indicator function c[t] at time t, defined as if the BLER is higher than the given threshold ε th , c[t] = 1, otherwise c[t] = 0. At the same time, the present application aims to maximize the number of eMBB symbols, and the reward function is defined as

[0090] ξ[t] = L max -l p [t] - l x [t] (1.7)

[0091] Since in the CMDP problem, the present application usually considers long-term rewards and long-term constraints. Therefore, under the deterministic policy μ, the eMBB long-term reward is given as follows:

[0092]

[0093] The URLLC long-term reliability constraint is expressed as follows:

[0094]

[0095] where Γ∈(0,1] is a discount factor.

[0096] Finally, as Figure 3 shown, the application proposes a CRPO-based TD3 algorithm, and the specific steps of the algorithm are as follows:

[0097] Step one: initialize the parameters of two sets of neural networks, including the action network set and the evaluation network set. The action network set is responsible for learning and feeding back the URLLC pilot length and data symbol number, denoted as the action vector a[t]=[l p [t],l x [t]],and the evaluation network set is responsible for evaluating the action vector;

[0098] Step two: according to the current real uplink channel gain a set of observation values According to the observation values, a sample transition space composed of the current state, the corresponding action (i.e. resource allocation strategy), the corresponding reward and cost, and the next state is obtained, denoted as <o[t],a[t],ξ[t],c[t],o[t+1]>,and stored in the experience replay memory.

[0099] Step three: repeat step two until the experience replay memory is filled, and execute step four;

[0100] Step four: randomly sample N s sets of transition sample data in the buffer, calculate the temporal-difference error (TD-error), and update the evaluation set network;

[0101] Step five: use the evaluation set network to calculate the URLLC long-term reliability constraint. If the constraint is violated, the action set network uses stochastic gradient descent to minimize the long-term reliability constraint, otherwise uses stochastic gradient ascent to maximize the long-term reward.

[0102] The application also discloses a resource allocation system based on coordinated multipoint transmission and deep reinforcement learning, which comprises a memory, a processor and a computer program stored in the memory, the computer program being configured to be called by the processor to realize the steps of the resource allocation method.

[0103] The application discloses a resource allocation method based on coordinated multipoint transmission and deep reinforcement learning, and the joint resource allocation problem of eMBB and URLLC is modeled into a constrained Markov decision problem by introducing the outage probability of URLLC block error rate (BLER). The number of pilot symbols and data transmission symbols allocated to URLLC users is dynamically regulated by the base station end every time slot, and the number of eMBB user symbols is maximized under the premise of meeting the reliability and delay constraints of URLLC, so that eMBB and URLLC users coexist.

[0104] The application further discloses a computer readable storage medium, which stores a computer program configured to realize the steps of the resource allocation method when called by a processor.

[0105] The application has the following advantages: 1. The application can not only meet the delay and reliability requirements of URLLC in a multi-cell network, but also meet the QoS indicators of URLLC in a high mobility scenario. At the same time, by dynamically regulating the URLLC pilot length and the number of data transmission symbols, the application can effectively reduce the degradation of eMBB user performance caused by puncturing; 2. When the channel estimation delay is within 2ms, the application can still effectively meet the 1ms delay and 99.9999% reliability requirements of URLLC users, while obtaining good eMBB performance.

[0106] The above is a further detailed description of the application in combination with specific preferred embodiments, and the specific implementation of the application should not be limited to these descriptions. For ordinary skilled persons in the technical field to which the application belongs, some simple deductions or substitutions can be made without departing from the concept of the application, and all of them should be regarded as falling within the protection scope of the application.

Claims

1. A resource allocation method based on cooperative multi-point transmission and deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Introduce the BLER interruption probability for URLLC users and model the joint resource allocation problem of eMBB and URLLC as a CMDP problem; Step 2: Employ a twin-delay deep deterministic policy gradient algorithm based on constraint correction strategy optimization. Through offline training, this algorithm can be used online to find the optimal resource allocation strategy with low complexity using forward propagation of a deep neural network, thereby reducing the processing latency of the algorithm. Step 1 includes the following steps: Step S1: Based on the imperfections of the channel's distinctness, the imperfection of the channel estimation delay, and the imperfection of the channel estimation error, the mathematical model of the imperfect channel is obtained, as shown below: （1.4） in ; Step S2: Calculate the signal-to-interference-plus-noise ratio (SIR) for URLLC users under imperfect channel conditions, as described below: （1.5） in This represents the impact of an imperfect channel. It is a precoded vector. This is an interference signal. It is Gaussian white noise; Step S3: Assume the short packet length of the URLLC data packet is... When the RRH cluster is URLLC packets are compiled into channel coding at all times. When there are 10 OFDM symbols, the achievable block error rate can be approximated by the short packet formula as follows: （1.6） in These are the subcarrier spacing and the duration of a single symbol, respectively. This is the actual transmission rate of the URLLC packet. It is the channel divergence; Step S4: Define the BLER interruption probability and model the CMDP problem; In step S1, imperfect channel heterogeneity: In a TDD system, the channel vectors for the uplink and downlink are modeled as follows: （1.1） in These represent the uplink and downlink times respectively. The channel vector, It characterizes the uncertainty of reciprocity in imperfect channels. Represents the channel distinct coefficients; Imperfections in channel estimation delay: In a CoMP system, the mathematical expression for the effect of channel estimation delay is as follows: （1.2） in At any moment The real channel, Indicates the channel estimation delay. It is a random vector. It is the channel correlation coefficient; Imperfections in channel estimation error: The channel estimation error is expressed as the difference between the actual channel and the estimated channel, and is described as follows: here Representative moment The estimated channel, Simultaneously, channel estimation error It is a random vector that follows a circularly symmetric complex Gaussian distribution, i.e. ,variance The mathematical form is as follows: （1.3） here and Representing time slots Path loss and RRH maximum transmission power, Represents noise variance. This indicates that in the time slot Pilot length allocated to URLLC users.

2. The resource allocation method according to claim 1, characterized in that, Step S4 further includes: Step 1: Give BLER violates indicator function at time Defined as Simultaneously, to maximize the number of eMBB symbols, the reward function is defined as: （1.7）； Step 2: In deterministic strategies The following is an explanation of how to obtain long-term eMBB rewards: （1.8）； The long-term reliability constraints of URLLC are expressed as follows: （1.9） in It is a conversion factor.

3. The resource allocation method according to claim 2, characterized in that, In step 1, if BLER is higher than a given threshold Indicator function Otherwise, the indicator function .

4. The resource allocation method according to claim 1, characterized in that, Step 2 also includes: Step 1: Initialize the parameters of the two sets of neural networks, including the action network set and the evaluation network set. The action network set is responsible for learning and feeding back the URLLC pilot length and the number of data symbols, denoted as the action vector. The evaluation network group is responsible for evaluating the action vector; Step 2: Based on the current actual uplink channel gain To obtain a set of observations Based on this observation, a sample transition space is obtained, consisting of the current state, corresponding action, corresponding reward and cost, and the next state, denoted as . And store it in the experience playback memory; Step 3: Repeat Step 2 until the experience playback memory is full, then proceed to Step 4; Step 4: Randomly sample from the buffer The evaluation group network is updated by transferring sample data from the group and calculating the temporal difference error. Step 5: Calculate the long-term reliability constraint of URLLC using the evaluation group network. If the constraint is violated, the action group network uses stochastic gradient descent to minimize the long-term reliability constraint; otherwise, it uses stochastic gradient ascent to maximize the long-term reward.

5. A resource allocation system based on cooperative multi-point transmission and deep reinforcement learning, characterized in that: include: A memory, a processor, and a computer program stored on the memory, the computer program being configured to implement the steps of the resource allocation method according to any one of claims 1-4 when invoked by the processor.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to implement the steps of the resource allocation method according to any one of claims 1-4 when invoked by a processor.

Images