RAN slice hierarchical radio resource management method based on multiple time scales

Abstract

The invention relates to a multi-time scale-based RAN (Radio Access Network) slice hierarchical wireless resource management method, which comprises the following steps of: allowing a service model 1 and a service model 2 to transmit on a resource block by utilizing an orthogonal multiple access punching mechanism and a punching mechanism consisting of a power domain non-orthogonal multiple access technology and a spatial domain non-orthogonal multiple access technology; the user of the service model 1 and the user of the service model 2 can utilize the wireless resource more efficiently or share the same wireless resource. On the basis, the invention provides a combination of a high-level control model and an optimal bottom-level control model to form a multi-time-scale wireless resource management strategy, and the strategy is also combined with deep reinforcement learning, so that the multi-time-scale wireless resource management is realized. The problem of solving a target function can be converted into a Gaussian Markov decision process by using an option-approver framework on a large time scale, and different service requirements correspond to sub-targets.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a RAN slice hierarchical wireless resource management method based on multiple time scales. Background Art

[0002] With the advent of the Internet of Everything era, many vertical service businesses are emerging, and the requirements of these businesses for quality of service (QoS) are extremely diverse. Therefore, the developing network slicing technology has come into being. The developing network slicing technology allows multiple logical networks to be virtualized on the same physical network to meet the specific needs of different users and applications. In this technology, each slice is dedicated to a specific type of business.

[0003] The main services of network slicing can be divided into three categories: ultra-reliable low-latency communication (uRLLC), enhanced mobile broadband (eMBB) and massive machine type communication (mMTC). The provision of these services requires different network data rates, reliability and latency.

[0004] However, how to effectively manage and coordinate multiple network slices in the radio access network (RAN) has become a major challenge, and the network slices in current 5G communications cannot effectively utilize the diverse quality of service (QoS) configuration technologies of limited and constrained resources. In the scenario where ultra-reliable low-latency communications and enhanced mobile broadband services coexist, there may be service conflicts caused by shared resource blocks during the transmission process. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to effectively manage and coordinate multiple network slices in a radio access network (RAN). In order to solve the above technical problem, the present invention provides a RAN slice hierarchical wireless resource management method based on multiple time scales.

[0006] The present invention provides a multi-time scale-based RAN slice hierarchical radio resource management method, comprising the following steps:

[0007] S1: In the scenario where two service models coexist, the total available bandwidth of the RAN is divided into multiple resource blocks, and the duration of a single time slot and the number of mini-time slots contained in each time slot are determined;

[0008] S2: At the initial moment of the current time slot, resource blocks are allocated to the service model 1 according to the resource requirements of the service model 1;

[0009] S3: Activation time of service model 2, judging whether there is free resource block;

[0010] If yes, proceed to the next step;

[0011] If not, execute step S5;

[0012] S4: Allocate unused resource blocks to the users requiring the service model 2, and return to execute step S2 after the next time slot;

[0013] S5: In a mini-time slot after the activation of the service model 2, an orthogonal multiple access puncturing mechanism is adopted to allow the service model 2 to be transmitted preferentially, or a superposition mechanism consisting of a power domain non-orthogonal multiple access technology and a space domain non-orthogonal multiple access technology is adopted to allow the service model 1 and the service model 2 to be transmitted simultaneously on the same resource block, and then the next step is executed;

[0014] S6: constructing a high-level control model to be optimized, and optimizing the parameters of the high-level control model to be optimized by using the option-critic strategy of reinforcement learning to obtain a high-level control model;

[0015] The high-level control model is configured to obtain a channel state and a user service flow change in each time slot through a channel model and a communication scenario, and use the channel state and the user service flow change to obtain a slice adjustment decision for setting priorities for users of the service model 1 and users of the service model 2;

[0016] S7: obtaining the slice adjustment decision through the high-level control model, constructing an optimized low-level control model corresponding to the slice adjustment decision, and solving the optimized low-level control model to obtain an optimal solution;

[0017] S8: Perform wireless resource scheduling based on the optimal solution through the base station to complete the data transmission of the service model 1 and the service model 2 in the current time slot, and return to execute step S2 after the next time slot.

[0018] The RAN slice hierarchical wireless resource management method based on multiple time scales disclosed in the present invention, in view of the technical problem of the present invention, utilizes the punching mechanism of orthogonal multiple access and the punching mechanism composed of power domain non-orthogonal multiple access technology and space domain non-orthogonal multiple access technology to allow business model 1 and business model 2 to be transmitted on one resource block, which can achieve more efficient use of wireless resources or sharing of the same wireless resources for users of business model 1 and users of business model 2. On this basis, the present invention proposes a combination of a high-level control model and an optimized low-level control model to form a multi-time scale wireless resource management strategy, and the strategy is also combined with deep reinforcement learning, which can transform the problem of solving the objective function into a Gaussian Markov decision process on a large time scale using the option-critic framework, and correspond different business needs to sub-goals. Through the scheme proposed by the present invention, the RAN network slice can make more intelligent and efficient wireless resource allocation according to the specific network status, and improve the adaptability and stability of the network slice to the Internet of Things scenario.

[0019] In a possible implementation manner, in step S6, the process of constructing the high-level control model to be optimized is as follows:

[0020] Construct a state space for describing the channel state and user service flow changes in each time slot;

[0021] Constructing an action space for representing the performance of selecting the slice of the business model 1 and the reliability of the slice of the business model 2;

[0022] Construct a reward function whose instantaneous value reflects the performance of the slice and whose expected cumulative value reflects the adaptive adjustment of the slice under different business traffic;

[0023] The high-level control model corresponding to this solution can simultaneously meet the performance of business model 1 and the reliability of business model 2, and can reduce the complexity of the network to a certain extent.

[0024] In a possible implementation, the state space is expressed as follows:

[0025]

[0026] In the formula,

[0027] ε represents the set of users of the business model 1;

[0028] A user set representing the business model 2;

[0029] s e,b,t Indicates that resource block b is allocated to user e of the service model 1 at time slot t;

[0030] Indicates that resource block b is allocated to user u of service model 2 at mini-time slot m of time slot t;

[0031] Indicates whether user e and user u adopt the hole punching mechanism;

[0032] represents the superposition of power domain non-orthogonal multiple access of user e and user u;

[0033] Indicates whether user e and user u use the superposition of non-orthogonal multiple access in the spatial domain;

[0034] h u,b,t represents the channel estimation value of user u in resource block b;

[0035] h e,b,t represents the channel estimation value of user e in resource block b;

[0036] The state space corresponding to this scheme can more accurately describe the channel state and user service flow changes in each time slot, which helps the high-level control model to obtain slice adjustment decisions for setting priorities for users of business model 1 and users of business model 2.

[0037] In a possible implementation, the action space is expressed as follows:

[0038] a=[e pr ,u pr ,s e,b,t ],

[0039] In the formula,

[0040] e pr ∈[0,1],u pr ∈[0,1]; e pr =1 indicates that the slice in this time period focuses on protecting the transmission rate of the service model 1, u pr =1 indicates that the slice in this time period focuses on the reliability of users of the service model 2; if both are 1 at the same time, a compromise scheduling method is adopted;

[0041] s e,b,t Indicates that the high-level control model allocates resource blocks to users of the service model 1 according to the state space at time slot t;

[0042] The action space corresponding to this solution helps the high-level control model to select the performance of the slice of business model 1 and the reliability of the slice of business model 2, so as to obtain a more accurate slice adjustment decision.

[0043] In a possible implementation, the reward function is calculated as follows:

[0044] r=k1r u +k2r net +k3r e ,

[0045]

[0046]

[0047] In the formula,

[0048] δ represents the overall packet loss rate of a mini-time slot;

[0049] It represents the maximum tolerable network complexity in RAN;

[0050] r e,t represents the data transmission rate of user e in the service model 1 at time slot t;

[0051] r e,min Indicates the minimum transmission rate of the service model 1;

[0052] The instantaneous value of the reward function of this scheme reflects the performance of the slice, and its expected cumulative value reflects the adaptive adjustment of the slice under different business traffic, which helps to optimize the parameters of the high-level control model through reinforcement learning.

[0053] In a possible implementation manner, the optimized underlying control model is expressed as follows:

[0054]

[0055] In the formula,

[0056] λ,ν are constants, which are selected based on the priority of the business model 1 and the business model 2;

[0057] In the optimization model of this solution, C1 and C2 ensure the performance isolation of the same service, C3 ensures that service model 1 and service model 2 can only use one multiplexing technology, and C4 and C5 respectively ensure the real-time power and transmission rate of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flow chart of a RAN slice hierarchical radio resource management method based on multiple time scales disclosed in an embodiment of the present application;

[0059] Figure 2 This is the scheduling of eMBB and uRLLC services in this embodiment. DETAILED DESCRIPTION

[0060] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0061] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] As shown in Figure 1, the embodiment of the present application discloses a RAN slice hierarchical radio resource management method based on multiple time scales, including the following steps:

[0063] S1: In the scenario where two service models coexist, the total available bandwidth of the RAN is divided into multiple resource blocks, and the duration of a single time slot and the number of mini-time slots contained in each time slot are determined.

[0064] Specifically, the RAN in this embodiment has a typical downlink cellular network system with a single base station, and the total available bandwidth is divided into B resource blocks. In these two service models, service model 1 is eMBB and service model 2 is uRLLC. In order to ensure the slicing performance of eMBB, the transmission of users is strictly guaranteed in this embodiment. That is, maximizing

[0065]

[0066] In the formula,

[0067] r e,t represents the rate of eMBB user e at time slot t;

[0068] Indicates the maximum rate of the defined user e.

[0069] At the same time, in order to ensure the reliability and low latency of the uRLLC slice, this embodiment defines that the uRLLC user u sends a fixed-size data packet, which must be transmitted within a mini-time slot during the base station transmission process, otherwise the data packet will be discarded. The packet loss of the uRLLC user is as follows:

[0070]

[0071] In the formula,

[0072] represents the transmission rate of the uRLLC user at the mini-time slot m of time slot t;

[0073] N u It represents the fixed data packet size of uRLLC. The formula ensures that the data packet of uRLLC is transmitted within one micro-time slot.

[0074] Figure 1 Indicates the scheduling of eMBB and uRLLC services. One frame is 1ms and contains 14 OFDM symbols (15kHz). One frame is set to one time slot, and one time slot is divided into multiple mini-slots. The scheduling of eMBB services is performed on each time slot, and the scheduling of uRLLC services is performed on each mini-slot.

[0075] S2: At the beginning of the current time slot, resource blocks are allocated to service model 1 according to the resource requirements of service model 1.

[0076] S3: Activation time of service model 2, judging whether there is free resource block;

[0077] If yes, proceed to the next step;

[0078] If not, execute step S5.

[0079] S4: Allocate unused resource blocks to users with business model 2 requirements, and then return to step S2 after the next time slot.

[0080] S5: In a micro-time slot after the activation of service model 2, an orthogonal multiple access puncturing mechanism is used to allow service model 2 to be transmitted first, or an overlay mechanism consisting of power domain non-orthogonal multiple access technology and space domain non-orthogonal multiple access technology is used to allow service model 1 and service model 2 to be transmitted simultaneously on the same resource block, and then the next step is executed.

[0081] When considering performance, we must also pay attention to the complexity of the network. When the channel gain difference between overlapping users is too small, the receiver needs more power to decode the overlapping users. Or when the spatial correlation is lower, the network complexity is lower. The network complexity is shown in the following formula, which can be expressed as the slice complexity introduced by the puncturing and superposition mechanisms respectively.

[0082]

[0083] In the formula,

[0084] Indicates whether user e and user u adopt the hole punching mechanism;

[0085] represents the superposition of power domain non-orthogonal multiple access of user e and user u;

[0086] Indicates whether user e and user u use the superposition of non-orthogonal multiple access in the spatial domain;

[0087] h u,b,t represents the channel estimation value of user u in resource block b;

[0088] he,b,t is the channel estimation value of user e in resource block b.

[0089] S6: Construct a high-level control model to be optimized, and optimize the parameters of the high-level control model to be optimized using the option-critic strategy of reinforcement learning to obtain the high-level control model. The high-level control model is set to obtain the channel state and user service flow changes in each time slot through the channel model and the communication scenario, and use the channel state and user service flow changes to obtain the slice adjustment decision for setting priorities for users of business model 1 and users of business model 2.

[0090] The high-level control model of this embodiment will select two modes for slices when making high-level decisions: radio resource allocation tends to eMBB slices and radio resource allocation tends to uRLLC slices. For example, when transmitting important high-definition video streams, the controller prefers eMBB services to ensure the stability and continuity of the video; while in the process of autonomous driving, in order to protect the safety of the vehicle, reliable operation must be completed in a short time, so uRLLC services must be given the first priority.

[0091] Specifically, in step S6, the process of constructing the high-level control model to be optimized is as follows:

[0092] First, a state space is constructed to describe the channel state and user service flow changes in each time slot.

[0093] The state space is an overall description of the channel state and service flow, and is the key to ensuring the effectiveness of the algorithm. The state space established in this embodiment is expressed as:

[0094]

[0095] In the formula,

[0096] ε represents the set of eMBB users, which is the cache status of eMBB users at time slot t;

[0097] represents the user set of uRLLC, which is the activated set of uRLLC devices at time slot t and micro-time slot m; s e,b,t It indicates that resource block b is allocated to user e of eMBB at time slot t;

[0098] It indicates that resource block b is allocated to user u of uRLLC at mini-slot m of time slot t.

[0099] Secondly, an action space is constructed to represent the performance of slices selected for business model 1 and the reliability of slices selected for business model 2.

[0100] The action space represents the selection of the performance of the eMBB slice and the reliability of the uRLLC slice. In view of the fact that the high-level control model is to decide the scheduling mode of the wireless resources, this embodiment sets the action vector of the high-level control model as the priority scheduling of the slice for eMBB and uRLLC at the current moment, and the corresponding action space is expressed as follows:

[0101] a=[e pr ,u pr ,s e,b,t ],

[0102] In the formula,

[0103] e pr ∈[0,1],u pr ∈[0,1],e pr =1 means that the slice in this time period focuses on protecting the transmission rate of eMBB, u pr =1 means that the slice focuses on the reliability of uRLLC users in this time period. If both are 1 at the same time, a compromise scheduling method is adopted. Different action selections will affect the user's perception function of the service in the lower layer model;

[0104] s e,b,t It indicates that the high-level control model allocates resource blocks to eMBB users according to the state space at time slot t.

[0105] Finally, a reward function is constructed whose instantaneous value reflects the quality of the slice performance, and whose expected cumulative value reflects the adaptive adjustment of the slice under different business traffic.

[0106] The instantaneous value of the reward function reflects the performance of the slice, and the expected cumulative value of the reward function reflects the adaptive adjustment of the slice under different business traffic. On this basis, this embodiment designs a combination of a long-term reward function and a short-term reward function to form a reward function.

[0107] 1. Short-term reward function

[0108] The short-term reward function mainly refers to the reliability of the uRLLC service. Specifically, once the uRLLC service is activated, it is necessary to immediately allocate resource blocks to it within the micro-timeslot. The reliability of the service can be measured by the packet loss rate. If the uRLLC service fails to complete data transmission within the specified micro-timeslot, this situation will be regarded as packet loss. Therefore, the reliability of the activated uRLLC service instance can be analyzed by calculating the total packet loss rate of the uRLLC service at a specific micro-time. Specifically, at any micro-timeslot, the packet loss rate of uRLLC cannot exceed the maximum tolerable packet loss probability in the RAN slice, that is,

[0109]

[0110] δ represents the overall packet loss rate of the mini-time slot, which can be obtained by the following formula based on the total number of activated uRLLC devices:

[0111]

[0112] In addition, the multiplexing of eMBB and uRLLC must be considered at each micro-time slot, and the introduction of the superposition / puncturing mechanism will increase the complexity of the network. When the channel gain difference of the superimposed users is too small, the receiving end needs more power to decode the overlapping users, or when the spatial correlation is lower, the network complexity is lower. To ensure the low network complexity of each micro-time slot, the difficulty of wireless resource management of the slice can be easily guaranteed. Therefore, in order to ensure the feasibility of the slice, this embodiment also designs a reward function for network complexity:

[0113]

[0114] Indicates the maximum tolerable network complexity in the RAN.

[0115] (II) Long-term reward function

[0116] In order to ensure that slices have relatively fair rewards for different services. This embodiment further designs a reward function for the eMBB transmission rate. Since its training time scale is the sum of m micro-timeslots, a larger reward feedback needs to be designed. The transmission rate of eMBB is affected by the uRLLC at each micro-timeslot:

[0117]

[0118] In the formula,

[0119] represents the signal to interference and noise ratio of user e at mini-slot m of time slot t in resource block b.

[0120] Since the eMBB service must ensure its transmission performance, this embodiment designs a minimum rate limit for each eMBB. When the eMBB slice meets the rate requirement, an additional reward is obtained at time slot t. Define the reward function at each time slot:

[0121]

[0122] Note that even users belonging to the same category have inherent service requirements, such as the minimum transmission rate r of eMBB. e,min , may also be different. Therefore, the high-level control model reward function is:

[0123] r=k1ru +k2r net +k3r e ,

[0124] r u It is the uRLLC business reward in the short-time reward function;

[0125] r net Rewards for the network;

[0126] For different user priority scheduling strategies, the convergence and reliability of training can be ensured by setting different reward weights.

[0127] In step S6, in the process of optimizing the parameters of the high-level control model using the option-critic strategy of reinforcement learning, the option-critic (OC) structure is used to ensure that the high-level control model makes the optimal slice adjustment decision in different environments, ensuring flexibility in time scale, that is, allocating wireless resources in each micro-time slot and determining the service priority of the next time period on a flexible and variable long time scale.

[0128] The option-critic structure optimizes the internal policy of the option and the termination network according to the policy gradient principle to maximize the discounted expected return. In the option-critic structure, the policy gradient within the option and the policy gradient of the termination condition are mainly concerned. Since this embodiment sets the model within the option as a convex optimization problem rather than a Markov decision process, only the policy gradient of the option termination condition needs to be considered here.

[0129] The gradient of the termination network of the high-level control model can be expressed as:

[0130]

[0131] Where;

[0132] θ β Indicates the parameters of the termination network;

[0133] η represents the thinking cost parameter introduced to prevent frequent switching of options;

[0134] A(s,o) represents the advantage function of the termination network;

[0135] Q Ω (s,o) represents the option value function, which is updated using the following formula:

[0136]

[0137] The network updates are as follows:

[0138]

[0139] The training process based on the option-critic structure is as follows: the channel state and service flow at the current moment are taken as input, the Q network is used to output the Q value of each slice scheduling behavior, and the wireless resource scheduling criterion with the largest Q value is selected; the Q network is updated according to formula (1), and then the termination network is updated according to formula (2), and the output is the termination probability of each slice strategy. If the slice strategy at the current moment is terminated at the next moment, the slice strategy with the largest Q value is re-selected using the Q network at the next moment.

[0140] Specifically, this embodiment uses a deep neural network with a parameter of θ to represent the long-time scale optimization strategy πe, and uses a real-time resource allocation objective function to represent the short-time scale optimization strategy πu. Because the lower layer is a convex optimization problem, it can be guaranteed that the slicing strategy on the short time scale is stable and convergent. On this basis, the upper layer model can be trained using a deep neural network to make πe converge.

[0141] S7: Obtain a slice adjustment decision through a high-level control model, construct an optimal low-level control model corresponding to the slice adjustment decision, and solve the optimal low-level control model to obtain an optimal solution.

[0142] In order to deeply analyze the selection of radio resource scheduling mode in the slice adjustment decision obtained by the high-level control model, this embodiment particularly considers the traffic characteristics and service modes of different QoS categories in eMBB and uRLLC services. To this end, different utility models are formulated in this embodiment to quantitatively measure the service satisfaction level of UE.

[0143] (I) QoS categories to ensure transmission performance

[0144] For enhanced services (e.g., IP voice transmission and adaptive video streaming), due to the user's inherent and preferred service requirements, the user's performance gain mainly depends on the resource allocation quality and is therefore susceptible to fluctuations, so the strategy tends to focus on the service quality of eMBB.

[0145] (II) QoS category focusing on reliability and low latency

[0146] For real-time, mission-critical and demanding applications (e.g. autonomous driving, vehicle platooning), a consistent and guaranteed level of service provision is required.

[0147] In the parameter optimization process of the high-level control model, the priority of the slice service has been obtained through the option-critic structure, so that the base station can reasonably and differently adjust its resource allocation strategy to adapt to the real-time network load, dynamic resource conditions and various needs of users. Different utility functions are proposed for the action selection of the high-level control model:

[0148]

[0149] Service model 1 represents the service of eMBB, and service model 2 represents the service of uRLLC. This embodiment further proposes an objective function p0 for allocating wireless resources at mini-time slots:

[0150]

[0151] Among them, C1 and C2 ensure the performance isolation of the same service, C3 ensures that eMBB and uRLLC can only use one multiplexing technology, and C4 and C5 ensure the real-time power and transmission rate of users respectively. The selection of λ and ν is mainly affected by the priority of eMBB and uRLLC.

[0152] The above optimization model is a deterministic convex optimization problem, and there must be a point where the constraints are not equal, so the Slater condition is satisfied. Therefore, we use the dual decomposition and KKT optimal conditions to solve the original convex problem. Now let's process the objective function:

[0153]

[0154] like is the optimal solution, then p * The following constraints must be satisfied and there exists a multiplier λ * =(λ1 * ,λ2 * ,λ3 * ) satisfies the following formula:

[0155]

[0156] λ * 1p * k,b =0

[0157]

[0158] λ * 1,λ * 2,λ * 3≥0

[0159] Because the original problem is a convex problem, the point that satisfies the KKT condition is also the original and dual optimal solution. This method can be used to allocate power to users in real time at micro-time slots, and the power allocation satisfies the characteristics of stability.

[0160] S8: Perform wireless resource scheduling based on the optimal solution through the base station to complete the data transmission of service model 1 and service model 2 in the current time slot, and return to execute step S2 after the next time slot.

[0161] The RAN slice hierarchical wireless resource management method based on multiple time scales disclosed in this embodiment uses a puncturing mechanism of orthogonal multiple access and a puncturing mechanism composed of power domain non-orthogonal multiple access technology and space domain non-orthogonal multiple access technology to allow service model 1 and service model 2 to be transmitted on one resource block, which can achieve more efficient use of wireless resources or sharing of the same wireless resources for users of service model 1 and users of service model 2. On this basis, this embodiment proposes a combination of a high-level control model and an optimized low-level control model to form a multi-time scale wireless resource management strategy, and the strategy is also combined with deep reinforcement learning, which can use the option-critic framework on a large time scale to convert the problem of solving the objective function into a Gaussian Markov decision process, and correspond different service requirements to sub-goals. Through the solution proposed by the present invention, the RAN network slice can make more intelligent and efficient wireless resource allocation according to the specific network status, and improve the adaptability and stability of the network slice to the Internet of Things scenario.

[0162] In the description of the embodiments of the present application, it should be noted that in the description of the present application, terms such as "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is only for the convenience of description, and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present application.

[0163] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0164] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A multi-time scale based RAN slice hierarchical radio resource management method, characterized in that: The steps include: S1: In the scenario where two service models coexist, the total available bandwidth of the RAN is divided into multiple resource blocks, and the duration of a single time slot and the number of mini-time slots contained in each time slot are determined; S2: At the initial moment of the current time slot, resource blocks are allocated to the service model 1 according to the resource requirements of the service model 1; S3: Activation time of service model 2, judging whether there is free resource block; If yes, proceed to the next step; If not, execute step S5; S4: Allocate unused resource blocks to the users requiring the service model 2, and return to execute step S2 after the next time slot; S5: In a mini-time slot after the activation of the service model 2, an orthogonal multiple access puncturing mechanism is adopted to allow the service model 2 to be transmitted preferentially, or a superposition mechanism consisting of a power domain non-orthogonal multiple access technology and a space domain non-orthogonal multiple access technology is adopted to allow the service model 1 and the service model 2 to be transmitted simultaneously on the same resource block, and then the next step is executed; S6: constructing a high-level control model to be optimized, and optimizing the parameters of the high-level control model to be optimized by using the option-critic strategy of reinforcement learning to obtain a high-level control model; The high-level control model is configured to obtain a channel state and a user service flow change in each time slot through a channel model and a communication scenario, and use the channel state and the user service flow change to obtain a slice adjustment decision for setting priorities for users of the service model 1 and users of the service model 2; S7: obtaining the slice adjustment decision through the high-level control model, constructing an optimized low-level control model corresponding to the slice adjustment decision, and solving the optimized low-level control model to obtain an optimal solution; S8: Perform wireless resource scheduling based on the optimal solution through the base station to complete the data transmission of the service model 1 and the service model 2 in the current time slot, and return to execute step S2 after the next time slot.

2. The multi-time-scale based RAN slice hierarchical radio resource management method according to claim 1, characterized in that: In step S6, the process of constructing the high-level control model to be optimized is as follows: Construct a state space for describing the channel state and user service flow changes in each time slot; Constructing an action space for representing the performance of selecting the slice of the business model 1 and the reliability of the slice of the business model 2; The instantaneous value is constructed to reflect the performance of the slice, and the expected cumulative value is constructed to reflect the reward function of the slice's adaptive adjustment under different business traffic.

3. The multi-time-scale based RAN slice hierarchical radio resource management method according to claim 2, characterized in that: The state space is expressed as follows: In the formula, ε represents the set of users of the business model 1; A user set representing the business model 2; s e,b,t Indicates that resource block b is allocated to user e of the service model 1 at time slot t; Indicates that resource block b is allocated to user u of service model 2 at mini-time slot m of time slot t; Indicates whether user e and user u adopt the hole punching mechanism; represents the superposition of power domain non-orthogonal multiple access of user e and user u; Indicates whether user e and user u use the superposition of non-orthogonal multiple access in the spatial domain; h u,b,t represents the channel estimation value of user u in resource block b; h e,b,t represents the channel estimation value of user e in resource block b.

4. The multi-time-scale based RAN slice hierarchical radio resource management method according to claim 3, characterized in that: The action space is expressed as follows: a=[and pr ,u pr ,s e,b,t ], In the formula, e pr ∈[0,1],u pr ∈[0,1]; e pr =1 indicates that the slice in this time period focuses on protecting the transmission rate of the service model 1, u pr =1 indicates that the slice in this time period focuses on the reliability of users of the service model 2; if both are 1 at the same time, a compromise scheduling method is adopted; s e,b,t It indicates that the high-level control model allocates resource blocks to users of the service model 1 according to the state space at time slot t.

5. The multi-time-scale based RAN slice hierarchical radio resource management method according to claim 4, characterized in that: The formula of the reward function is as follows: r=k1r u +k2r net +k3r e , In the formula, δ represents the overall packet loss rate of a mini-time slot; It represents the maximum tolerable network complexity in RAN; r e,t represents the data transmission rate of user e in the service model 1 at time slot t; r e,min Indicates the minimum transmission rate of the service model 1.

6. The multi-time-scale based RAN slice hierarchical radio resource management method according to claim 5, characterized in that: The optimal underlying control model is expressed as follows: In the formula, λ,ν are constants, and their selection is based on the priority of the business model 1 and the business model 2.

Images