A method for intelligent optimization of RAN slicing and MAC scheduling in combination

Abstract

The application discloses a RAN slice and MAC scheduling joint intelligent optimization method, which is applied to a multi-slice multi-user uplink communication scene and at least comprises: constructing a multi-time scale hierarchical architecture model with the goal of maximizing long-term system utility; constructing a large time scale deep reinforcement learning to train a slice-level RB resource allocation strategy, a new reward clipping method to guarantee convergence performance in a lightweight system, and a traffic perception module to guide experience pool sampling to accelerate convergence; constructing a small time scale deep reinforcement learning for each slice to train a slice-differentiated MAC scheduler parameter strategy; each slice allocates RB resources to users according to a scheduling priority at each scheduling time slot; and the double time scale deep reinforcement learning is alternately trained until convergence. The application makes full use of the optimization capability of reinforcement learning in a complex system, significantly improves the SLA satisfaction rate of different types of slices and system spectrum efficiency, and effectively realizes a multi-slice multi-user communication system with high system utility, light weight and high robustness.

Description

Technical Field

[0001] This invention belongs to the fields of network slicing technology and intelligent communication technology, specifically relating to a joint intelligent optimization method for RAN slicing and MAC scheduling. Background Technology

[0002] In 5G and future 6G wireless communications, service scenarios are becoming increasingly diverse, and heterogeneous applications with differentiated Service Level Agreements (SLAs) requirements are constantly emerging, such as 8K high-definition video transmission, telemedicine, VR / AR, and smart cities. In the 3GPP standard, these are categorized into three typical cases: enhanced mobile broadband (eMBB), ultra-reliable low-latency communication (URLLC), and massive machine-type communication (mMTC). Traditional mobile networks serving a single service cannot adapt to the flexible and diverse service requirements of 5G / 6G, and building different physical networks for different services is impractical. Therefore, in order to build networks with different service types and different SLA requirements on the same physical network infrastructure, network slicing based on Software Defined Networking (SDN) has emerged.

[0003] Network slicing is typically divided into core network slices, bearer network slices, and radio access network (RAN) slices. On the RAN side, due to the scarcity of bandwidth resources and the dynamic changes in radio channels, traditional static and quasi-static resource allocation methods are inefficient and cannot meet the SLA requirements of each service scenario. Therefore, a dynamic RAN slice resource allocation strategy is needed. Deep Reinforcement Learning (DRL) trains an agent to continuously interact with the environment, generating intelligent and dynamic slice configurations and resource scheduling schemes. It adaptively adjusts according to environmental changes, thereby optimizing long-term resource utilization to meet the SLA requirements of each slice. Therefore, it is very suitable for solving the RAN slice resource allocation problem.

[0004] Existing RAN slicing technologies based on DRL (Device Streaming Restriction) often employ a hierarchical resource allocation approach due to the high algorithmic complexity of directly allocating bandwidth to users with different SLA (Service Level Agreement) requirements via Resource Blocks (RBs). There are two main approaches: First, DRL is used to allocate total resources to each slice, with traditional MAC (Macro-Assisted Mapping) scheduling algorithms such as round-robin within each slice. This method effectively allocates resources across slices; however, traditional scheduling algorithms cannot guarantee resource utilization in dynamic environments. Second, DRL is first used to allocate resources to slices, and then another DRL algorithm is used to directly allocate resources to users. This method effectively guarantees the SLA requirements of each user within a slice; however, it skips the MAC layer scheduler and directly allocates resources to users, which differs from the priority-based scheduling method in current communication networks, making it difficult to directly apply to existing communication networks. Summary of the Invention

[0005] The purpose of this invention is to utilize deep reinforcement learning technology to take the weighted sum of SLA satisfaction rate and spectral efficiency of each slice as the overall system utility, and jointly optimize the resource allocation between RAN slices and the hyperparameters of the MAC layer scheduler within each slice. This not only improves the SLA requirements of different slices and different users within the same slice in dynamic environments while ensuring spectral efficiency, but also allows for the use of the MAC layer scheduler for inter-slice scheduling, making it easy to apply in existing communication networks. Furthermore, this solution optimizes the traditional DRL algorithm by using reward clipping to make the network structure more lightweight, thereby reducing system complexity; and adds a traffic-aware module to improve system robustness, enabling the designed algorithm to quickly learn and transfer in different traffic change scenarios, providing a joint intelligent optimization method for RAN slices and MAC scheduling.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a joint intelligent optimization method for RAN slicing and MAC scheduling, comprising:

[0007] S1: Construct a RAN slicing system model with a multi-time-scale hierarchical architecture in a multi-slice, multi-user uplink transmission scenario. The optimization objective is to maximize the long-term system utility, i.e., the weighted sum of the SLA satisfaction rate and spectral efficiency of each slice.

[0008] S2: Construct deep reinforcement learning on a large time scale in the RAN slice system model with a multi-time-scale hierarchical architecture, train and generate slice-level RB resource allocation strategy, and pre-allocate RB resources to N slices, where the large time scale specifically refers to the time scale with the slice time window as the smallest time unit.

[0009] S3: In the RAN slice system model with multi-timescale hierarchical architecture, construct deep reinforcement learning for each slice at a small timescale, train and generate differentiated MAC layer scheduler hyperparameter tuning strategies for each slice, adjust the scheduling priority of each slice's MAC layer scheduler, and the maximum number of RBs that can be allocated to each user. Here, a small timescale refers to a timescale with the hyperparameter tuning window as the smallest time unit.

[0010] S4: Each slice, based on its differentiated MAC layer scheduler hyperparameters, calls its own independent MAC layer scheduler to allocate RB resources to users belonging to that slice in each scheduling time slot according to the scheduling priority of its own MAC layer scheduler.

[0011] S5: Alternately train large-scale deep reinforcement learning and small-scale deep reinforcement learning for each slice until convergence to optimize the slice bandwidth pre-allocation scheme generated by large-scale deep reinforcement learning. n (f)} and the hyperparameter tuning scheme for the MAC layer scheduler generated by deep reinforcement learning for each slice at a small time scale. n (f, g), where ξ n (f, g) = {α n (f, g), β n (f, g), z n (f, g)}.

[0012] Furthermore, the RAN slicing system model with a multi-time-scale hierarchical architecture in step S1 includes: N slices and U users. Users access the slices, and the slices provide the same type of service to the users accessing them. The same type of service includes the same latency requirements and the same rate requirements.

[0013] Furthermore, a slice time window consists of G consecutive hyperparameter tuning windows, and a hyperparameter scheduling window consists of C scheduling time slots T.

[0014] Furthermore, the optimization objective in step S1 is specifically expressed as maximizing long-term system utility:

[0015]

[0016]

[0017] Among them, w n (f) represents the bandwidth allocated to slice n within the f-th slice time window, SSR n(f) represents the SLA satisfaction rate of slice n within the f-th slice time window. The system utility is the weighted sum of the slice SLA satisfaction rate (SSR) and the system frequency efficiency (SE). The slice SLA satisfaction rate is the proportion of the number of packets transmitted by slice users that meet the SLA requirements within a slice time window to the total number of packets transmitted by slice users. SE(f) represents the spectral efficiency within the f-th slice time window, W represents the total system bandwidth, and b1, ..., b n denoted by , where 'a' represents the adjustable weights of each slice SSR, and 'a' represents the system frequency efficiency SE.

[0018] Further, step S2 specifically includes:

[0019] S21: Constructing deep reinforcement learning algorithms on large time scales, including establishing the state space. Action space and reward r L ;

[0020] S22: Transfer the state space Establish the number of data packets required for each slice system to achieve utility and successful transmission while meeting SLA requirements, i.e.

[0021]

[0022] Among them, SSR n (f) represents the SLA satisfaction rate of slice n within the f-th slice time window, and SE(f) represents the spectral efficiency within the f-th slice time window. This represents the number of data packets that have been transmitted in slice n and have met the SLA requirements within the f-th slice time window;

[0023] S23: Move space Establish pre-allocated bandwidth combinations for each slice, i.e.

[0024]

[0025] Where w n (f) represents the bandwidth allocated to slice n within the f-th slice time window;

[0026] S24: Set u L The set of segmentation points consists of several constants. And reward r L The design is for a function r of J(f). L (f) At the same time, a new reward editing method is introduced, which is characterized by when The reward is a continuous function, when The time reward is a discrete function, and the final reward function is r. L (f) Specifically

[0027]

[0028] in, The coefficients of the continuous reward function represent the system utility within the f-th slice time window. Discrete reward value

[0029] Furthermore, step 2 introduces a flow sensing module. When a significant change in the flow of any slice is detected, the module guides the near-end of the experience pool to prioritize sampling. The specific process includes:

[0030] A1: Set traffic awareness threshold ∈ L ;

[0031] A2: Obtain and calculate the average flow rate within I slice time windows.

[0032]

[0033] in, This represents the number of users in slice n. Indicates user u within the f-th slice time window m，n The number of packets transmitted;

[0034] A3: When the flow change of any slice exceeds the threshold, i.e. At that time, deep reinforcement learning on a large time scale prioritizes sampling data after the f-th slice time window.

[0035] Furthermore, step S3 specifically includes:

[0036] S31: Construct deep reinforcement learning algorithms for each slice at small time scales, including establishing the state space of slice n. Action space of slice n And the reward of slice n

[0037] S32: Slice the state space of n The average number of packets transmitted, average latency, and average bandwidth for slice n users within a hyperparameter tuning window are established.

[0038]

[0039] in, This represents the number of users in slice n. This represents the user u within the f-th slice time window and the g-th hyperparameter tuning window. m，n Number of packets transmitted, l m，n (f, g) represents user u within the f-th slice time window and the g-th hyperparameter tuning window. m，nThe total latency, i.e., the sum of the latencies of all data packets in the user's buffer queue, w n (f) represents the bandwidth of slice n within the f-th slice time window;

[0040] S33: The action space of slice n Construct the hyperparameters ξ for the MAC layer scheduler n The (f, g) combination, i.e.

[0041]

[0042] Where, ξ n (f, g) = {α n (f, g), β n (f, g), z n (f, g)} is discretized into {k} β k α k z} values, K = k β ×k α ×k z ;

[0043] S34: Set u S The set of segmentation points consists of several constants. And reward r S The design is for a function r of J(f, g). S (f, g), and at the same time introduce a new reward clipping method, which is characterized by when The reward is a continuous function, when The time reward is a discrete function, and the final reward function is r. S (f, g) specifically means

[0044]

[0045] Among them, J n (f, g) = b n ×SSR n (f, g) + a × SE(f, g) represents the utility of slice n within the g-th hyperparameter tuning window of the f-th slice time window, and is the coefficient of the continuous reward function. Discrete reward value

[0046] Furthermore, in step S4, in each scheduling time slot, a slice bandwidth pre-allocation scheme {w} generated by large-scale deep reinforcement learning is applied. n (f) and hyperparameter tuning scheme for the MAC layer scheduler generated by small-scale deep reinforcement learning ξ n (f, g) = {α n (f, g), β n(f, g), z n Each slice (f, g) is independently scheduled using its own MAC layer scheduler. The scheduling steps for slice n are as follows:

[0047] S41: Calculate the scheduling priority of users within slice n.

[0048]

[0049] Among them, l m，n (t0) and For user u m，n The total delay of the packet to be transmitted at the start of time slot t0 and the average transmission rate of the successfully transmitted packet in the previous time slot;

[0050] S42: Calculate user u m，n The expected bandwidth, i.e., user u m，n The bandwidth required for all waiting packets to be transmitted in that time slot. in Indicates user u m，n The total number of bits of all packets in the buffer at time t0. This indicates that at time t0, each RB can be used for user u. m，n The average rate provided;

[0051] S43: Calculate the pre-allocated bandwidth for users within slice n.

[0052]

[0053] S44: Initialize the remaining bandwidth resources of slice n The total bandwidth of the slice w n (f), where w n (f) is the RB bandwidth w RB The integer multiple of P, and initialize the current user as P. m，n The largest user, with a scheduling order of P m，n Descending order;

[0054] S45: Allocate current user u m，n The actual bandwidth is

[0055]

[0056] S46: Update the remaining bandwidth of slice n

[0057] S47: Determine if there is remaining bandwidth If all slices n users have been assigned, the process ends; otherwise, the current user is updated to the next user according to the scheduling order, and the process jumps to step S45.

[0058] Furthermore, step S5 specifically includes:

[0059] S51, Large-scale deep reinforcement learning iterative training e L During this period, deep reinforcement learning at small timescales only makes decisions and collects experiences (s, a, r, s') into its experience base, but does not train;

[0060] S52, Small-scale deep reinforcement learning iterative training e S During this period, deep reinforcement learning on a large timescale only makes decisions and collects experience (s, a, r, s') into its experience base, but does not train;

[0061] S53, repeating S51 and S52, online training and decision-making for deep reinforcement learning at large time scales and for each slice at small time scales, until convergence, to optimize the slice bandwidth pre-allocation scheme {w n (f)} and MAC layer scheduler hyperparameter tuning scheme ξ n (f, g) = {α n (f, g), β n (f, g), z n (f, g)}.

[0062] Beneficial Effects: Compared with existing technologies, the technical solution of this invention has the following beneficial effects: The proposed RAN slicing and MAC scheduling joint intelligent optimization method utilizes the optimization capabilities of deep reinforcement learning in complex systems. In complex scenarios with multiple slices and multiple users, it designs a multi-time-scale hierarchical architecture to achieve RAN slice bandwidth resource pre-allocation and MAC layer scheduler hyperparameter joint tuning, significantly improving the SLA satisfaction rate of slices of different service types and the overall system spectral efficiency. Novel reward clipping makes the network more lightweight without sacrificing performance. Traffic awareness increases robustness and enhances the adaptability of this method under different traffic environments. Attached Figure Description

[0063] Figure 1 This is a flowchart of a joint intelligent optimization method for RAN slicing and MAC scheduling in an embodiment of the present invention;

[0064] Figure 2 This is a comparison diagram of the system utility of a joint intelligent optimization method for RAN slicing and MAC scheduling in an embodiment of the present invention and a traditional method;

[0065] Figure 3 This is a comparison diagram of the system utility of reward clipping, continuous reward, and discrete reward in an embodiment of the present invention;

[0066] Figure 4 This is a comparison diagram of the system utility of flow-aware and non-flow-aware systems in an embodiment of the present invention. Detailed Implementation

[0067] The invention will now be further explained with reference to the accompanying drawings.

[0068] like Figure 1 As shown, this invention provides a joint intelligent optimization method for RAN slicing and MAC scheduling, comprising:

[0069] S1: Construct a RAN slicing system model with a multi-time-scale hierarchical architecture in a multi-slice, multi-user uplink transmission scenario. The optimization objective is to maximize the long-term system utility, i.e., the weighted sum of the SLA satisfaction rate and spectral efficiency of each slice.

[0070] S2: Construct deep reinforcement learning on a large time scale in the RAN slice system model with a multi-time-scale hierarchical architecture, train and generate slice-level RB resource allocation strategies, and pre-allocate RB resources to each slice. The large time scale specifically refers to the time scale with the slice time window as the smallest time unit.

[0071] S3: In the RAN slice system model with multi-timescale hierarchical architecture, construct deep reinforcement learning for each slice at a small timescale, train and generate differentiated MAC layer scheduler hyperparameter tuning strategies for each slice, adjust the scheduling priority of each slice's MAC layer scheduler, and the maximum number of RBs that can be allocated to each user. Here, a small timescale refers to a timescale with the hyperparameter tuning window as the smallest time unit.

[0072] S4: Each slice, based on its differentiated MAC layer scheduler hyperparameters, calls its own independent MAC layer scheduler to allocate RB resources to users belonging to that slice in each scheduling time slot according to the scheduling priority of its own MAC layer scheduler.

[0073] S5: Alternately train large-scale deep reinforcement learning and small-scale deep reinforcement learning for each slice until convergence to optimize the slice bandwidth pre-allocation scheme generated by large-scale deep reinforcement learning. n (f)} and the hyperparameter tuning scheme for the MAC layer scheduler generated by deep reinforcement learning for each slice at a small time scale. n (f, g), where ξ n (f, g) = {α n (f, g), β n (f, g), z n (f, g)}.

[0074] In step S1, in a multi-slice, multi-user uplink transmission scenario, there are N slices and U users. Each user can access one slice, and a slice provides the same type of service to all users within it; that is, users accessing the same slice have essentially the same SLA requirements. Multiple users access the base station via Orthogonal Frequency Division Multiple Access (OFDMA) mode. Let the user set be denoted as... Among them, u m，n This represents the m-th user in slice n. This represents the total number of users within slice n. The same type of service specifically refers to, but is not limited to, services with the same latency requirements and the same rate requirements.

[0075] The system operates in a time-slot manner, with each scheduling time slot denoted by T. S scheduling time slots T form a slice time window. A slice time window consists of G consecutive MAC layer scheduler hyperparameter tuning windows (referred to as hyperparameter tuning windows), and each hyperparameter scheduling window consists of C scheduling time slots T, i.e., G × C = S. Let (f) represent the f-th slice time window, (f, g) be the g-th hyperparameter adjustment window of the f-th slice window, and t0 = (fS + gC + t)T, representing the t-th scheduling time slot of the g-th hyperparameter adjustment window of the f-th slice time window. Then, at t0, the MAC layer scheduler's scheduling priority for users belonging to slice n is...

[0076]

[0077] Among them, l m，n (t0) and For user u m，n The total delay of packets to be transmitted at the start of time slot t0 and the average transmission rate of packets successfully transmitted in the previous time slot. α n (f, g) and β n (f, g) represent the values ​​of l within the g-th hyperparameter adjustment window of the f-th slice time window, respectively. m，n (t0) and The weighting coefficients.

[0078] remember Indicates user u at time t0 m，n The expected bandwidth, i.e., user u m，n The bandwidth required for all waiting packets to be transmitted in that time slot. in Indicates user u m，n The size of the packet in the buffer at time t0. This indicates that at time t0, each RB can be used for user u. m，nThe average bandwidth provided. Considering that a slice cannot be entirely occupied by one user, leading to compromised communication quality for other users, a maximum bandwidth that can be allocated to a user is included. Let z be the bandwidth allocated to the user. n (f, g) represents the maximum allocable bandwidth ratio of slice n within the g-th hyperparameter adjustment window of the f-th slice time window, i.e., the proportion of pre-allocated bandwidth w of slice n occupied by users belonging to slice n. n (f) is the maximum proportion; then at t0, user u m，n The pre-allocated bandwidth is

[0079]

[0080] In this step, the specific implementation scenario of the present invention is a multi-slice, multi-user uplink transmission scenario. The base station service area is a circular area with a radius of 40m. The system operates in the 2GHz band, with a receive gain of 20dB, MIMO spatial stream number of 4, and a total allocable bandwidth resource W = 80MHz. The large time-scale bandwidth resolution is 1MHz, and the bandwidth of each RB resource is w. RB =180kHz. The system has three service type slices, N=3, namely VoLTE slice, eMBB slice, and URLLC slice. Slices 1, 2, and 3 correspond to the number of users accessing each slice: 50, 20, and 30 respectively, using Orthogonal Frequency Division Multiple Access (OFDMA) mode. The system operates in time-slot mode, with each scheduling time slot denoted by T, where T=0.5ms. A slice time window is 1s, consisting of 2000 time slots. A hyperparameter tuning window is 0.1ms, consisting of 200 time slots, and a slice time window contains 10 consecutive hyperparameter tuning windows. That is, S=2000, C=200, G=10.

[0081] Furthermore, within the hierarchical RAN system framework, considering both RB resource pre-allocation and MAC layer scheduler hyperparameter tuning, specifically, the system operation modes at three time scales are as follows:

[0082] (1) On a large time scale, with the slice time window as the smallest time unit, at the beginning of each slice time window, RB resources are pre-allocated, and the RB resources with a total bandwidth of W are allocated to N slices, with slice n obtaining a bandwidth of w. n (f) The bandwidth pre-allocated to each slice remains unchanged within the slice's time window.

[0083] (2) On a small time scale, with the hyperparameter tuning window as the smallest time unit, at the beginning of each hyperparameter tuning window, perform independent hyperparameter adjustments for the MAC layer scheduler of each slice, including but not limited to the scheduling priority calculation formula coefficient α of the MAC layer scheduler of each slice. n (f, g), β n(f, g), and the maximum allocatable bandwidth ratio of the slice z n (f, g). Hyperparameters α of the MAC layer scheduler for each slice n (f, g), β n (f, g) and z n (f, g) are kept unchanged within this hyperparameter tuning window.

[0084] (3) In each scheduling slot, VoLTE, eMBB, and URLLC slices are independently scheduled using their respective MAC layer schedulers, in descending order of scheduling priority. n (f) allocates bandwidth to each user.

[0085] In a multi-timescale hierarchical RAN slicing system model, the condition for a user's packet to meet the SLA requirement is that both latency and rate requirements are simultaneously satisfied, i.e.

[0086]

[0087] in, Indicates user u m，n The data packet transmission rate, This represents the total latency of the data packet, including waiting time. and transmission delay Further definition Indicates user u m，n The packets simultaneously meet the rate and latency requirements. This indicates that the conditions cannot be met simultaneously or that the conditions cannot be met simultaneously.

[0088] Further defining the Slice SLA Satisfaction Rate (SSR) is the proportion of packets transmitted by a slice user within a slice time window that meet the SLA requirements, out of the total number of packets transmitted by the slice user.

[0089]

[0090] in, This indicates that user u is within the f-th slice time window. m，n The number of packets transmitted;

[0091] Similarly, the spectral efficiency is defined as

[0092]

[0093] in, For user u m，n The size of the transmitted packets, where W is the total system bandwidth;

[0094] Therefore, system utility is defined as the weighted sum of VoLTE, eMBB, URLLC slice SSR, and system frequency efficiency SE, i.e.

[0095]

[0096] Where a and b1, ..., b n These represent the adjustable weights of SE and each slice SSR, respectively.

[0097] The system optimization objective is to maximize long-term system efficiency through large-scale slice bandwidth pre-allocation and small-scale MAC layer scheduler hyperparameter adjustment.

[0098]

[0099]

[0100] In this embodiment, the SE weight coefficient a = 1 is set, and the SSR weight coefficients for VoLTE, eMBB, and URLLC are b1 = 1, b2 = 4, and b3 = 6, respectively.

[0101] In step S2, a deep reinforcement learning algorithm is constructed on a large time scale, including establishing the state space. Action space and reward r L The design of the reward function involves a novel reward clipping method to improve convergence capability, and introduces a traffic-aware module to guide the sampling of the experience pool, which accelerates convergence and improves system robustness when traffic changes are large.

[0102] state space Establish the number of data packets required for each slice system to achieve utility and successful transmission while meeting SLA requirements, i.e.

[0103]

[0104] Among them, SSR n (f) represents the SLA satisfaction rate of slice n within the f-th slice time window, and SE(f) represents the spectral efficiency within the f-th slice time window. This represents the number of data packets that have been transmitted in slice n and have met the SLA requirements within the f-th slice time window.

[0105] Action space Establish pre-allocated bandwidth combinations for each slice, i.e.

[0106]

[0107] Where w n (f) represents the bandwidth allocated to slice n within the f-th slice time window.

[0108] For reward r LThe design, setting u L The set of segmentation points consists of several constants. And reward r L The design is for a function r of J(f). L (f) At the same time, a new reward editing method is introduced, which is characterized by when The reward is a continuous function, when The time reward is a discrete function, and the final reward function is r. L (f) Specifically

[0109]

[0110] in, The coefficients of the continuous reward function represent the system utility within the f-th slice time window. Discrete reward value

[0111] Deep reinforcement learning on a large timescale determines the RAN slice bandwidth pre-allocation scheme and simultaneously updates the training neural network with current environmental data. The decision-making process involves selecting the action of the current state as the policy output according to the ε-Greedy principle, thus forming the pre-allocation bandwidth scheme for each slice. The ε-Greedy principle means that there is an ε probability of selecting a random action and a 1-ε probability of selecting the action with the largest Q-value in the current state. The Q-values ​​of each action in the current state are obtained through the neural network. In this embodiment, ε is a function of the number of iterations (epochs). The training process involves obtaining the system utility J(f) of the selected pre-allocated bandwidth scheme after one slice time window, and then applying the reward function r. L (f) Obtain rewards and add experiences in the form (s, a, r, s′) to the experience pool, where s represents the current state, a represents the action, r represents the reward, and s′ represents the next state. During training, randomly select several experiences from the experience pool to update the neural network.

[0112] In this step, N = 3. The 80MHz bandwidth is divided into 3 slices with a resolution of 1MHz, resulting in an action space of {(1, 1, 78), (1, 2, 77), ..., (78, 1, 1)}, totaling 3160 actions. L =1, The reward function is designed as follows

[0113] Step S2 of this invention also introduces a flow sensing module. When a large change in the flow of any slice is detected, it guides the near end of the experience pool to prioritize sampling. The specific process includes:

[0114] S21, Set the traffic awareness threshold ∈ L =0.3;

[0115] S22, Obtain and calculate the average flow rate within I = 20 slice time windows.

[0116]

[0117] in, This represents the number of users in slice n. Indicates user u within the f-th slice time window m，n The number of packets transmitted;

[0118] S23, when the flow change of any slice exceeds the threshold, i.e. At that time, deep reinforcement learning on a large time scale prioritizes sampling data after the f-th slice time window.

[0119] In step S3, deep reinforcement learning algorithms are constructed for VoLTE, eMBB, and URLLC slices at small time scales. The Dueling DQN algorithm is selected as the deep reinforcement learning algorithm, which includes establishing the state space. Action space and rewards And use the reward clipping method described in S2.

[0120] The state space of slice n The average number of packets transmitted, average latency, and average bandwidth for slice n users within a hyperparameter tuning window are established.

[0121]

[0122] in, This represents the number of users in slice n. This represents the user u within the f-th slice time window and the g-th hyperparameter tuning window. m，n Number of packets transmitted, l m，n (f, g) represents the latency user u within the f-th slice time window and the g-th hyperparameter tuning window. m，n Total latency, w n (f) represents the bandwidth of slice n within the f-th slice time window.

[0123] Action space of slice n The hyperparameters of the MAC layer scheduler are constructed as a combination, and considering the adjustability of the actual MAC layer scheduler, they are discretized. MAC layer scheduler hyperparameter ξ n (f, g) = {α n (f, g), β n (f, g), z n (f, g)} is discretized into {k}β k α k z There are} values, and the action space has a total of K values, where K = k β ×k α ×k z Therefore, the action space of slice n for

[0124] The reward for slice n Designed for J n The function (f, g) sets u S The set of segmentation points consists of several constants. At the same time, a novel reward clipping method similar to that described in S2 is introduced, characterized by when The reward is a continuous function, when The time reward is a discrete function, and the final reward function is r. S (f, g) specifically means

[0125]

[0126] Among them, J n (f, g) = b n ×SSR n (f, g) + a × SE(f, g) represents the utility of slice n within the *g*th hyperparameter tuning window of the *f*th slice time window, and is the coefficient of the continuous reward function. Discrete reward value

[0127] In deep reinforcement learning at small timescales, the decision-making process optimizes the hyperparameters of the MAC layer scheduler while simultaneously updating the training neural network with current environmental data. The decision-making process involves the deep reinforcement learning agent in slice n selecting the action of the current state as the policy output using the ε-Greedy principle, where ε is a function of the number of iterations (epochs). The training process involves selecting the MAC layer hyperparameter configuration scheme for slice n, and then, after a hyperparameter tuning window, obtaining the effect J of slice n. n (f, g), through the reward function Calculate the reward and add the experience in the form (s, a, r, s′) to the experience pool. During training, randomly select a number of experiences from the experience pool to update the neural network.

[0128] In this embodiment, it is converted to in, The value z takes the form of {100, 50, 20, 10, 5, 2, 1, 1 / 2, 1 / 5, 1 / 10, 1 / 20, 1 / 50, 1 / 100}. n (f, g) takes values ​​{0.1, 0.2, 0.3, ..., 1}, resulting in 130 possible actions. Therefore, the action space of slice n is... The reward function is specifically designed as follows:

[0129]

[0130] In step S4, in each scheduling time slot, VoLTE, eMBB, and URLLC slices are pre-allocated bandwidth according to the slice bandwidth pre-allocation scheme generated by large-scale deep reinforcement learning {w n (f)} and the MAC layer hyperparameter tuning scheme generated by small-scale deep reinforcement learning ξ n (f, g) = {α n (f, g), β n (f, g), z n Each of the (f, g) groups is independently scheduled using its respective MAC layer scheduler. The specific scheduling steps are as follows:

[0131] S41: Calculate the scheduling priority of users within slice n.

[0132]

[0133] Among them, l m，n (t0) and For user u m，n The total delay of the packet to be transmitted at the start of time slot t0 and the average transmission rate of the successfully transmitted packet in the previous time slot.

[0134] S42: Calculate user u m，n The expected bandwidth, i.e., user u m，n The bandwidth required for all waiting packets to be transmitted in that time slot. in Indicates user u m，n The size of the packet in the buffer at time t0. This indicates that at time t0, each RB can be used for user u. m，n The average rate provided.

[0135] S43: Calculate the pre-allocated bandwidth for users within slice n.

[0136]

[0137] S44: Initialize the remaining bandwidth resources of slice n The total bandwidth of the slice w n (f), where wn (f) is the RB bandwidth w RB Multiples of P; initialize the current user as P m，n The largest user, with a scheduling order of P m，n Descending order.

[0138] S45: Allocate current user u m，n The actual bandwidth is

[0139]

[0140] S46: Update the remaining bandwidth of slice n

[0141] S47: Determine if there is remaining bandwidth If all users in slice n are assigned, the process ends; otherwise, the current user is updated to the next user according to the scheduling order, and the process jumps to S45.

[0142] In step S5, reinforcement learning at both large and small timescales iterates as follows:

[0143] S51: Iterative Training of Deep Reinforcement Learning over Large Timescales L = 200 times, during which the deep reinforcement learning at small time scales only makes decisions and collects experience (s, a, r, s′) into its experience base, but does not train.

[0144] S52: Iterative Training of Deep Reinforcement Learning on Small Timescales S =1000 times, during which the deep reinforcement learning on a large time scale only makes decisions and collects experience (s, a, r, s′) into its experience base, but does not train.

[0145] S53: Repeat S51 and S52, performing online training and decision-making for deep reinforcement learning at large timescales and for each slice at small timescales until convergence, to optimize the slice bandwidth pre-allocation scheme. n (f)} and MAC layer scheduler hyperparameter tuning scheme ξ n (f, g) = {α n (f, g), β n (f, g), z n (f, g)} maximizes the system utility.

[0146] The embodiments of the present invention were simulated and verified under a Rayleigh channel, utilizing... Figure 2 The system utility of five RAN slicing and MAC layer joint scheduling methods was compared:

[0147] (1) Traffic-aware Twin Time Scale Dueling Deep Q Network (TTS-DDQN), which is an embodiment of the present invention, has a traffic-aware module and reward clipping. The Dueling DQN algorithm is used for slice bandwidth pre-allocation and MAC layer hyperparameter tuning.

[0148] (2) Twin Time Scale GAN Dueling Deep Q Network (TTS-GDQN) does not use reward clipping. Slice bandwidth pre-allocation and MAC layer hyperparameter tuning both use the GANDueling DQN algorithm.

[0149] (3) One Time Scale Dueling Deep Q Network (OTS-DDQN), with Dueling DQN used for slice bandwidth pre-allocation, and the MAC layer using the RR algorithm for scheduling.

[0150] (4) One Time Scale GAN Dueling Deep Q Network (OTS-GDQN), the slice bandwidth is pre-allocated using GAN Dueling DQN, and the MAC layer uses the RR algorithm for scheduling.

[0151] (5) Hard slices, the slice bandwidth is pre-allocated based on the ratio of historical traffic and latency rate requirements of each slice, and the MAC layer uses the RR algorithm for scheduling.

[0152] The above comparisons show that TTS-DDQN, TTS-GDQN, OTS-DDQN, and OTS-GDQN have higher system utility than hard slicing, indicating that the adaptive capability of deep reinforcement learning applied to slice bandwidth pre-allocation can effectively improve system utility. The TTS-DDQN scheme requires only 50.07% of the floating-point operations per second of TTS-GDQN, yet its system utility is superior, demonstrating that reward clipping can make the system more lightweight and efficient. Both TTS-DDQN and TTS-GDQN use deep reinforcement learning to jointly optimize slice bandwidth pre-allocation and MAC layer hyperparameters on a dual-timescale basis, and their system utility is significantly better than OTS-DDQN and OTS-GDQN, which only use deep reinforcement learning on a single timescale. This indicates that the SLA-oriented RAN slice resource allocation and MAC layer joint scheduling method based on deep reinforcement learning can effectively improve system utility and has versatility, adapting to different deep reinforcement learning algorithms.

[0153] The embodiments of the present invention utilize Figure 3 The study compared TTS-DDQN using reward clipping, continuous rewards, and discrete rewards, demonstrating that reward clipping combines the stability of continuous rewards with the fast convergence capability of discrete rewards, enabling the system utility to reach a high level and converge more quickly, thus showcasing the effectiveness of reward clipping.

[0154] This invention simulates a scenario where the flow rate of each slice changes drastically during the 10,000th iteration, utilizing... Figure 4 The comparison between TTS-DDQN with and without a traffic-aware module demonstrates that, under conditions of significant traffic fluctuations, the traffic-aware module guides the sampling of the experience pool, enabling the algorithm to reconverge more quickly, maintaining a high level of system utility, and enhancing system robustness.

[0155] This invention leverages the optimization capabilities of deep reinforcement learning in complex systems. In complex scenarios with multiple slices and multiple users, it designs a multi-time-scale hierarchical architecture to achieve RAN slice bandwidth resource pre-allocation and MAC layer scheduler hyperparameter joint tuning, significantly improving SLA satisfaction rates for slices of different service types and overall system spectral efficiency. A novel reward clipping mechanism makes the network more lightweight without sacrificing performance. Traffic awareness enhances robustness and improves the method's adaptability under different traffic environments. Therefore, it significantly improves SLA satisfaction rates for different slice types and system spectral efficiency, effectively realizing a high-utility, lightweight, and robust multi-slice, multi-user communication system.

[0156] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A joint intelligent optimization method for RAN slicing and MAC scheduling, characterized in that, include: S1: Construct a RAN slicing system model with a multi-time-scale hierarchical architecture in a multi-slice, multi-user uplink transmission scenario. The optimization objective is to maximize the long-term system utility, i.e., the weighted sum of the SLA satisfaction rate and spectral efficiency of each slice. S2: Construct deep reinforcement learning on a large time scale in the RAN slice system model with a multi-time-scale hierarchical architecture, train and generate slice-level RB resource allocation strategy, and pre-allocate RB resources to N slices, where the large time scale specifically refers to the time scale with the slice time window as the smallest time unit. S3: In the RAN slice system model with multi-timescale hierarchical architecture, construct deep reinforcement learning for each slice at a small timescale, train and generate differentiated MAC layer scheduler hyperparameter tuning strategies for each slice, adjust the scheduling priority of each slice's MAC layer scheduler, and the maximum number of RBs that can be allocated to each user. Here, a small timescale refers to a timescale with the hyperparameter tuning window as the smallest time unit. S4: Each slice, based on its differentiated MAC layer scheduler hyperparameters, calls its own independent MAC layer scheduler to allocate RB resources to users belonging to that slice in each scheduling time slot according to the scheduling priority of its own MAC layer scheduler. S5: Alternately train large-scale deep reinforcement learning and small-scale deep reinforcement learning for each slice until convergence to optimize the slice bandwidth pre-allocation scheme generated by large-scale deep reinforcement learning. n (f)} and the hyperparameter tuning scheme for the MAC layer scheduler generated by deep reinforcement learning for each slice at a small time scale. n (f,g), where ξ n (f,g)={α n (f,g),β n (f,g),z n (f,g)}.

2. The RAN slicing and MAC scheduling joint intelligent optimization method according to claim 1, characterized in that, The RAN slicing system model with a multi-time-scale hierarchical architecture in step S1 includes: N slices and U users. Users access the slices, and the slices provide the same type of service to the users accessing them. The same type of service includes the same latency requirements and the same rate requirements.

3. The RAN slicing and MAC scheduling joint intelligent optimization method according to claim 1, characterized in that, A slice time window consists of G consecutive hyperparameter tuning windows, and a hyperparameter tuning window consists of C scheduling time slots T.

4. The RAN slicing and MAC scheduling joint intelligent optimization method according to claim 1, characterized in that, The optimization objective in step S1 is specifically expressed as maximizing long-term system utility: Among them, w n (f) represents the bandwidth allocated to slice n within the f-th slice time window, SSR n (f) represents the SLA satisfaction rate of slice n within the f-th slice time window. The system utility is the weighted sum of the slice SLA satisfaction rate (SSR) and the system frequency efficiency (SE). The slice SLA satisfaction rate is the proportion of the number of packets transmitted by slice users that meet the SLA requirements within a slice time window to the total number of packets transmitted by slice users. SE(f) represents the spectral efficiency within the f-th slice time window, W represents the total system bandwidth, and b1,...,b n denoted by , where 'a' represents the adjustable weights of each slice SSR, and 'a' represents the system frequency efficiency SE.

5. The RAN slicing and MAC scheduling joint intelligent optimization method according to claim 1, characterized in that, Step S2 specifically includes: S21: Constructing deep reinforcement learning algorithms on large time scales, including establishing the state space. Action space and reward r L ; S22: Transfer the state space Establish the number of data packets required for each slice system to achieve utility and successful transmission while meeting SLA requirements, i.e. Among them, SSR n (f) represents the SLA satisfaction rate of slice n within the f-th slice time window, and SE(f) represents the spectral efficiency within the f-th slice time window. This represents the number of data packets that have been transmitted in slice n and have met the SLA requirements within the f-th slice time window; S23: Move space Establish pre-allocated bandwidth combinations for each slice, i.e. Where w n (f) represents the bandwidth allocated to slice n within the f-th slice time window; S24: Set u L The set of segmentation points consists of several constants. And reward r L The design is for a function r of J(f). L (f) At the same time, a new reward editing method is introduced, which is characterized by when The reward is a continuous function, when The time reward is a discrete function, and the final reward function is r. L (f) Specifically in, The coefficients of the continuous reward function represent the system utility within the f-th slice time window. Discrete reward value 6. The RAN slicing and MAC scheduling joint intelligent optimization method according to claim 1, characterized in that, Step 2 also introduces a traffic sensing module. When a significant change in traffic volume is detected in any slice, it guides the near-end of the experience pool to prioritize sampling. The specific process includes: A1: Set traffic awareness threshold ∈ L ; A2: Obtain and calculate the average flow rate within I slice time windows. in, This represents the number of users in slice n. Indicates user u within the f-th slice time window m,n The number of packets transmitted; A3: When the flow change of any slice exceeds the threshold, i.e. At that time, deep reinforcement learning on a large time scale prioritizes sampling data after the f-th slice time window.

7. The joint intelligent optimization method for RAN slicing and MAC scheduling according to claim 1, characterized in that, Step S3 specifically includes: S31: Construct deep reinforcement learning algorithms for each slice at small time scales, including establishing the state space of slice n. Action space of slice n And the reward of slice n S32: Slice the state space of n The average number of packets transmitted, average latency, and average bandwidth for slice n users within a hyperparameter tuning window are established. in, Represents the number of users in slice n This represents the user u within the f-th slice time window and the g-th hyperparameter tuning window. m,n Number of packets transmitted, l m,n (f,g) represents user u within the f-th slice time window and the g-th hyperparameter tuning window. m,n The total latency, i.e., the sum of the latencies of all data packets in the user's buffer queue, w n (f) represents the bandwidth of slice n within the f-th slice time window; S33: The action space of slice n Construct the hyperparameters ξ for the MAC layer scheduler n The (f,g) combination, i.e. Where, ξ n (f,g)={α n (f,g),β n (f,g),z n (f,g)} is discretized into {k β ,k α ,k z } values, K = k β ×k α ×k z ; S34: Set u S The set of segmentation points consists of several constants. And reward r S The design is for a function r of J(f,g). S (f,g), and introduces a new reward editing method, characterized by when The reward is a continuous function, when The time reward is a discrete function, and the final reward function is r. S (f,g) specifically means Among them, J n (f,g)=b n ×SSR n (f,g)+a×SE(d,g) represents the utility of slice n within the g-th hyperparameter tuning window of the f-th slice time window, and is the coefficient of the continuous reward function. Discrete reward value 8. The RAN slicing and MAC scheduling joint intelligent optimization method according to claim 1, characterized in that, In step S4, for each scheduling time slot, a slice bandwidth pre-allocation scheme {w} generated by large-scale deep reinforcement learning is applied. n (f) and hyperparameter tuning scheme for the MAC layer scheduler generated by small-scale deep reinforcement learning ξ n (f,g)={α n (f,g),β n (f,g),z n Each slice (f,g)} is independently scheduled using its own MAC layer scheduler. The scheduling steps for slice n are as follows: S41: Calculate the scheduling priority of users within slice n. Among them, l m,n (t0) and For user u m,n The total delay of the packet to be transmitted at the start of time slot t0 and the average transmission rate of the successfully transmitted packet in the previous time slot; S42: Calculate user u m,n The expected bandwidth, i.e., user u m,n The bandwidth required for all waiting packets to be transmitted in that time slot. in Indicates user u m,n The total number of bits of all packets in the buffer at time t0. This indicates that at time t0, each RB can be used for user u. m,n The average rate provided; S43: Calculate the pre-allocated bandwidth for users within slice n. S44: Initialize the remaining bandwidth resources of slice n The total bandwidth of the slice w n (f), where w n (f) is the RB bandwidth w RB The integer multiple of P, and initialize the current user as P. m,n The largest user, with a scheduling order of P m,n Descending order; S45: Allocate current user u m,n The actual bandwidth is S46: Update the remaining bandwidth of slice n S47: Determine if there is remaining bandwidth If all slices n users have been assigned, the process ends; otherwise, the current user is updated to the next user according to the scheduling order, and the process jumps to step S45.

9. The joint intelligent optimization method for RAN slicing and MAC scheduling according to claim 1, characterized in that, The S5 step specifically includes: S51, Large-scale deep reinforcement learning iterative training e L During this period, deep reinforcement learning at small timescales only makes decisions and collects experiences (s,a,r,s') into its experience base, but does not train; S52, Small-scale deep reinforcement learning iterative training e S During this period, deep reinforcement learning on a large timescale only makes decisions and collects experience (s,a,r,s') into its experience base, but does not train; S53, repeating S51 and S52, online training and decision-making for deep reinforcement learning at large time scales and for each slice at small time scales, until convergence, to optimize the slice bandwidth pre-allocation scheme {w n (f)} and MAC layer scheduler hyperparameter tuning scheme ξ n (f,g)={α n (f,g),β n (f,g),z n (f,g)}.

Images