A slice-aware dynamic resource allocation method based on deep reinforcement learning

Abstract

This invention relates to the field of communication network technology, specifically to a slice-aware dynamic resource allocation method based on deep reinforcement learning. By constructing a system model that integrates wireless channel characteristics and multi-slice service requirements, the differentiated quality of service requirements for the coexistence of eMBB and URLLC are formalized as a joint optimization problem aimed at maximizing system service satisfaction. Through the design of slice-aware states, actions, and rewards, the method achieves coordinated dynamic allocation of time-domain, frequency-domain, and power resources. This solves the problem that in existing technologies, when considering resource allocation in the time, frequency, and power domains simultaneously, the action space dimension is often too high, leading to difficulties in model training, slow algorithm convergence, and potentially the curse of dimensionality. The method achieves adaptive allocation of multi-service resources in a dynamic network environment, balancing spectral efficiency, user fairness, and differentiated quality of service assurance.

Description

Technical Field

[0001] This invention relates to the field of communication network technology, and more specifically to a slice-aware dynamic resource allocation method based on deep reinforcement learning. Background Technology

[0002] With the large-scale commercialization of 5G and the gradual clarification of the future vision of 6G, mobile communication networks are increasingly penetrating all aspects of social production and daily life. The rise of emerging business scenarios such as augmented reality (AR), virtual reality (VR), industrial internet, autonomous driving, and telemedicine has placed diversified, differentiated, and even extreme service quality requirements on network performance: some services require enhanced mobile broadband (eMBB), while others require ultra-reliable low latency communication (URLLC).

[0003] To meet the differentiated needs of Enhanced Mobile Broadband (eMBB) and Ultra-Reliable and Low-Latency Communications (URLLC) services in 5G networks, network slicing technology divides physical network resources into multiple logical slices as needed, enabling customized services and QoS (Quality of Service) guarantees in heterogeneous service scenarios. However, within the 5G network slicing framework, eMBB and URLLC services share the same physical network infrastructure, yet they exhibit significant resource competition: eMBB services prioritize high bandwidth, while URLLC services require low latency and high reliability.

[0004] Deep Reinforcement Learning (DRL) represents a paradigm shift from "manually designed rules" to "autonomous learning by agents." Through continuous interaction with the environment, DRL agents can directly master resource allocation strategies that achieve optimal long-term performance in dynamic, stochastic network environments without requiring precise mathematical models or complex manual rules. Therefore, DRL has been widely adopted in research on network slice resource allocation.

[0005] To improve the performance of DRL in complex networks, its agent architecture is constantly evolving. Multi-Agent Deep Reinforcement Learning (MADRL) is used for distributed decision-making to improve scalability, while hierarchical DRL improves learning efficiency by decomposing the problem. Furthermore, Graph Neural Networks (GNNs) are introduced to model complex topological relationships between users, Long Short-Term Memory (LSTM) networks are used for business prediction to provide forward-looking decisions, and transfer learning is used to accelerate policy convergence in new environments. However, in typical scenarios where eMBB and URLLC coexist, existing research focuses too much on improving agent architecture and input features, neglecting to address core decision-making mechanisms such as state, action, and reward functions to achieve precise adaptation to the differentiated service quality requirements of different service segments. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a slice-aware dynamic resource allocation method based on deep reinforcement learning. By designing slice-aware states, actions, and reward functions, it achieves performance superior to traditional methods and some complex DRL variants, providing a simpler solution for efficient dynamic resource allocation.

[0007] The technical solution adopted by this invention to solve its technical problem is: A slice-aware dynamic resource allocation method based on deep reinforcement learning is applicable to scenarios where eMBB and URLLC services coexist in 5G and future networks. This method deploys a pre-trained deep reinforcement learning agent on the base station side. By continuously sensing the network state, it dynamically and collaboratively adjusts time slot, frequency domain, and power domain resources, thereby improving overall system efficiency and service quality. The method includes the following steps: Step 1: Identify the resource allocation optimization issues for the combined eMBB and URLLC services; Step 2: Construct the resource allocation optimization problem in Step 1 into a Markov Decision Process (MDP) model, and identify its core elements, which include the agent, state space, action space, and reward function. Step 3: Design a dual-delay deep deterministic strategy gradient algorithm for slice awareness. The algorithm relies on the TD3 framework to jointly optimize time and frequency resources and transmit power, realize dynamic adaptation to network slice requirements, and achieve adaptive allocation of multi-service resources in dynamic network environment.

[0008] Furthermore, step 1 includes the following steps: The overall user experience quality (QoE) is quantified as system service satisfaction (S), specifically defined as the proportion of users whose communication needs are met out of the total number of users. This indicator considers both the throughput requirements of eMBB services and the reliability and latency requirements of URLLC services. Its mathematical expression is as follows: ; Among them, M eMBB and M URLLC The total number of users in the two categories, respectively; and For binary indicator variables: when the throughput requirements of eMBB user m are met, Otherwise, it is 0; when the latency and reliability requirements of URLLC user m are simultaneously met, Otherwise, it is 0; the specific service quality requirements are given by the following formula: ; in, r e , ε u and t u These represent the actual throughput of eMBB users and the actual transmission error rate and latency of URLLC users, respectively. r s e , ε s u and t s u This corresponds to the service quality threshold, namely the minimum rate required for eMBB, the maximum error rate allowed by URLLC, and the maximum latency.

[0009] The resource allocation optimization problem for combining eMBB and URLLC services is expressed as: ; ; ; ; Among them, P m P is the transmit power allocated by the serving base station to user m; max η is the maximum transmit power of the system. m This indicates the time-domain preemption ratio of URLLC users over eMBB users. x k m Indicates whether resource block k has been allocated to user m: ; .

[0010] Furthermore, step 2 includes: In the MDP model, the agent acts as the decision-making body, perceives the network environment by observing the state, and performs actions accordingly to implement resource allocation. The environment then transitions to a new state based on the actions performed and provides a reward signal to evaluate the immediate benefits of the actions. Through a closed-loop interaction consisting of state, action, and reward, the intelligent agent gradually learns a dynamic resource allocation strategy that maximizes long-term cumulative returns.

[0011] Furthermore, in step 2, the intelligent agent is defined as follows: the intelligent agent is the central controller of a single base station, responsible for making unified resource allocation decisions for wireless network scenarios including both eMBB and URLLC service users within its coverage area. This intelligent agent achieves dynamic optimization and allocation of spectrum resources by observing the global network status and outputting continuous control actions.

[0012] Furthermore, in step 2, the state space is defined as: the state of the system at each moment. S t It is a high-dimensional vector that comprehensively represents the real-time operating status of the network, specifically including the information-to-dryness ratio γ for each user. m and data type identifier for each user Type m ( Type m =1 indicates the URLLC type. Type m = 0 indicates eMBB type). The mathematical expression of the state vector is: S t = {γ m , Type m}

[0013] Furthermore, in step 2, the action space is defined as: the time slot in which each agent... t The action set consists of key decision variables in the resource allocation optimization problem of joint eMBB and URLLC services, and is defined as a three-dimensional continuous action vector. a ( t )= { α rb , β pw , η}. In the action set α rbThe quantity represents the adjustment coefficient for allocating resource blocks based on an average allocation. β pw This represents the adjustment factor for the transmission power based on the average distribution. η This represents the proportion of time slot resources that URLLC services seize from eMBB services. First, an average allocation baseline for resource blocks and power is calculated based on the number of users, and then fine-tuning is achieved through adjustments in the action vector. α rb and β pw The definition range is between 0 and 2, and the micro-slot preemption ratio is... η In actual execution, it is quantized into discrete values ​​[0, 2 / 14, 4 / 14, 7 / 14, 1.0] to match the 5G frame structure.

[0014] Furthermore, in step 2, the reward function is defined as: designing a multi-objective weighted reward function. R ( t ): R ( t ) = w urllc × R urllc +w eMBB × R eMBB + w efficiency × R efficiency ; in, R ( t This represents an immediate reward at time step t, designed to simultaneously optimize the differentiated quality of service requirements of eMBB and URLLC services. w urllc , w eMBB , w efficiency All are weighting coefficients.

[0015] The reward formula for URLLC services is: ; ; ; Where, ε s ε is the system's tolerance threshold for error rate. u For real-time error rate, t s t is the system's tolerance time delay threshold. u For real-time latency, werror As the weight of the error rate, w delay The weights are used to account for latency; when the service requirement threshold is exceeded, a logarithmic function is used to avoid training instability caused by extreme values.

[0016] eMBB service reward formula: ; in, This indicates the percentage of eMBB users who meet the service requirements. To meet the eMBB rate requirements of users, M eMBB This represents the total number of eMBB users. This incentive aims to improve overall service satisfaction for eMBB slices by maximizing the proportion of eMBB users meeting rate requirements, thereby promoting fairness in resource allocation among users.

[0017] R efficiency To evaluate preemption efficiency based on the Spearman rank correlation coefficient, and to encourage users to preempt time slot resources from those with ample frequency domain resources or good channel quality, it is expressed as: ; ; ; in, ρ 1. ρ 2 is the rank correlation coefficient. ρ 1. Calculate the correlation between eMBB user resource allocation and preemption ratio. ρ 2. Calculate the correlation between signal-to-noise ratio and preemption ratio.

[0018] Weighting coefficient w urllc , w eMBB , w efficiency , w error and w delay The method was determined through experimental optimization and is used to balance the relative importance of different business objectives.

[0019] r ratio The ratio used to measure the actual rate of return to the demand rate is expressed as: ; ; ; in, r actualIt is the actual rate value calculated based on resource blocks, power, and micro-slot preemption. r require The required rate is calculated based on the current data volume, where η is the time slot preemption ratio. x k m This indicates whether resource block k has been allocated to user m. f k The bandwidth of a resource block. γ m Let D be the signal-to-interference-plus-noise ratio (SIR) for user m, D be the channel dispersion, and c be the block length. v For error rate, B u To serve the amount of data transmitted by users, τ Duration of URLLC service.

[0020] R guidance To measure whether the average micro-slot preemption ratio is reasonable, it is set as a piecewise function, expressed as: ; in, W guidance To maximize the weight of the guidance and reward portions, penalties are imposed when the allocation is insufficient or excessive, while rewards are given when the demand is just met. r ratio It is the ratio of the actual rate to the required rate.

[0021] Furthermore, the slice-aware dual-delay deep deterministic policy gradient algorithm SA-TD3 adopts an Actor-Critic architecture, the core framework of which is: an agent (base station) observes the network state. S t The extracted preemptive reward features are used to output continuous actions via the Actor network. a t This includes resource blocks, power, and preemption ratio; two independent Critic networks are used to evaluate the long-term value of this state-action pair. Q 1( S t , a t ), Q 2( S t , a t The smaller value is taken as the target Q value to reduce overestimation bias and guide the policy update of the Actor network. At the same time, an experience replay buffer is created to store the interaction data between the agent and the environment. The introduction of the experience replay mechanism ensures the stability of the learning process.

[0022] Furthermore, the training process of the SA-TD3 is as follows: First, the Actor network is randomly initialized. With two Critic networks , The parameters, These are the parameters of the neural network. These parameters are then copied to their respective target networks. , and To provide a stable learning objective, an experience replay buffer R of size N is created to store the agent's interaction data with the environment, and a stochastic process N (typically Gaussian noise) is initialized to encourage exploration early in training. Key hyperparameters are also set, including the policy update frequency d, the variance σ of the target policy's smoothing noise, and the action clipping range. And the soft update coefficient τ, etc.

[0023] Furthermore, the SA-TD3 training is performed cyclically across multiple episodes. Within each episode, the agent (i.e., the base station central controller) first observes the current network state. This state is a composite vector containing the information-to-dryness ratio for each user. and business type identifier (1 represents URLLC, 0 represents eMBB). Next, the Actor network outputs a basic 3D continuous action vector based on this state. .in, It is the adjustment factor for resource block allocation. These are the adjustment coefficients for transmit power, and both range from 0 to 2. η represents the proportion of time slots that URLLC services preempt from eMBB services, which is quantized into discrete values ​​{0, 2 / 14, 4 / 14, 7 / 14, 1.0} to match the 5G frame structure during actual execution. To facilitate exploration, noise will be added to the basic operations. And crop the final result: The base station then performs this action, allocating resources according to the adjusted plan. The environment then comprehensively assesses the impact of this resource allocation on multiple objectives, including eMBB throughput, URLLC latency, and reliability, and calculates and feeds back an immediate scalar reward. This reward is the core guiding signal for algorithm learning; it quantifies the immediate effect of the current action on meeting the quality of service requirements of heterogeneous slices. The environment then provides an immediate reward. and the next state The experience tuple of this interaction ( It is stored in the experience replay buffer.

[0024] Furthermore, once the amount of data in the buffer reaches the preset batch size, the network update step is initiated, which specifically includes: First, randomly sample a batch of N experience samples from the buffer; during the update process, the reward... r t It is a key input for calculating the target Q value, and it directly determines the baseline direction for state-action pair value estimation; The Critic network's updates are guided by calculating the target's Q-value: first, smooth noise is added to the target action, i.e. , The noise follows a Gaussian distribution; then, the smaller value of the two Critic target network outputs is taken to calculate the target. Where γ is the discount factor; then, by minimizing the Q-value estimate of the current Critic network relative to the target... The mean squared error loss between the two is used to update the parameters of the Critic network.

[0025] The Actor network employs a delayed update strategy, executing once every d steps. Its update direction follows the policy gradient that maximizes the Q-value, which is the long-term value assessment learned by the Critic network based on historical rewards. Therefore, rewards indirectly guide the optimization direction of the Actor policy by influencing the Q-value. This gradient is jointly determined by the gradient of the Critic network with respect to actions and the gradient of the Actor network with respect to parameters.

[0026] Finally, all target networks slowly track the parameters of their corresponding main networks using soft updates: , , where τ is a coefficient much smaller than 1, ensuring the stability of the learning process.

[0027] The entire training process described above is repeated until the preset number of rounds is reached or performance convergence is achieved. Through these steps, the SA-TD3 algorithm can autonomously learn a resource allocation strategy that simultaneously guarantees high throughput for eMBB and low latency and high reliability for URLLC under time-varying channels and burst traffic, guided by reward signals and through continuous interaction with the dynamic environment. Its dual-critic structure and delayed update mechanism effectively alleviate the overestimation problem of the value function, improving the stability and final performance of the algorithm.

[0028] Technical effects of the present invention: Compared with existing technologies, the slice-aware dynamic resource allocation method based on deep reinforcement learning of the present invention has the following advantages: 1. This invention constructs a system model that integrates wireless channel characteristics and multi-slice service requirements, and formalizes the differentiated quality of service requirements of eMBB and URLLC coexisting into a joint optimization problem with the goal of maximizing system service satisfaction; 2. This invention proposes a deep reinforcement learning framework based on dual-delay deep deterministic policy gradient (TD3). Through slice-aware state, action, and reward design, it achieves coordinated dynamic allocation of time-domain, frequency-domain, and power resources. This solves the problem that in the design of action space in existing technologies, if the resource allocation in the time-domain, frequency-domain, and power-domain is considered simultaneously, the dimensionality of the action space is often too high, leading to difficulties in model training, slow algorithm convergence, and the potential curse of dimensionality. In addition, it innovatively constructs the preemptive guidance reward as a multi-scale temporal feature, which is embedded into the Actor input through a dedicated network, significantly enhancing the agent's temporal insight and decision-making accuracy. 3. Through system simulation, this invention compares the proposed algorithm with various benchmark methods such as static allocation and DQN-based methods in multiple service scenarios. The results show that the proposed algorithm outperforms existing methods in terms of spectrum utilization, user fairness, transmission reliability, and latency performance, verifying its effectiveness and superiority in actual deployment. Attached Figure Description

[0029] Figure 1 This is a flowchart of the slice-aware dynamic resource allocation method based on deep reinforcement learning according to the present invention. Figure 2 This is a diagram of the SA-TD3 algorithm framework of the present invention; Figure 3 This is a diagram illustrating the training process of the present invention; Figure 4 This is a graph showing the average rate of eMBB service in this invention. Figure 5 This is a latency diagram for the URLLC service of this invention; Figure 6 This is a transmission error rate diagram for the URLLC service of this invention; Figure 7 This is a comparison chart of the total rate in different business scenarios of the present invention; Figure 8 This is a comparison chart of spectrum utilization in different business scenarios of the present invention; Figure 9 This is a comparison chart of eMBB service rates for different services according to the present invention; Figure 10 Comparison chart of transmission latency for different URLLC services; Figure 11 Comparison of transmission error rates for URLLC services of different business types; Figure 12 This is a comparison chart of the spectral utilization rates of different algorithms in this invention; Figure 13 This is a comparison chart of eMBB service rates for different algorithms of this invention; Figure 14 This is a comparison chart of the fairness index of eMBB services using different algorithms of this invention; Figure 15 This is a comparison chart of the error rates of different URLLC services using the algorithms of this invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0031] Example 1: like Figure 1 As shown in the figure, this embodiment relates to a slice-aware dynamic resource allocation method based on deep reinforcement learning, which is applicable to the coexistence scenario of eMBB and URLLC services in 5G and future networks. This method deploys a pre-trained deep reinforcement learning agent on the base station side. By continuously sensing the network status, it dynamically and collaboratively adjusts time slot, frequency domain, and power domain resources, thereby improving the overall system efficiency and service quality. Specifically, it includes the following steps: Step 1: Identify the resource allocation optimization issues for the combined eMBB and URLLC services; Step 2: Construct the resource allocation optimization problem in Step 1 into a Markov Decision Process (MDP) model, and identify its core elements, which include the agent, state space, action space, and reward function. Step 3: Design a dual-delay deep deterministic strategy gradient algorithm for slice awareness. The algorithm relies on the TD3 framework to jointly optimize time and frequency resources and transmit power, realize dynamic adaptation to network slice requirements, and achieve adaptive allocation of multi-service resources in dynamic network environment.

[0032] 1. Description of the resource allocation optimization problem combining eMBB and URLLC services in this embodiment In the dynamic slicing system model, all user terminals are represented by M={1,…,M}. eMBB users send traffic at a fixed rate, while URLLC traffic arrival follows a Poisson distribution, with the Poisson arrival rate represented by… λ Representation. Let K denote the set of resource blocks (RBs) K = {1, 2, ..., k, ...}, where each resource block k ∈ K contains frequency domain information. f k The bandwidth and the duration of one TTI in the time domain. x This indicates whether each resource block k is allocated to a specific user. A resource block can only be allocated to one user, meaning the resource has exclusivity. The formula is as follows: ; ; in, t Indicates the sequence number of the TTI time slot.

[0033] A complete timeslot contains 14 symbols of time, while microtimeslots can be divided into 2, 4, or 7 symbols of time. η m The formula representing the time-domain preemption ratio of URLLC users over eMBB users is as follows: ; In networks employing Orthogonal Multiple Access (OMA) technology, the system provides services to different users through orthogonal resource blocks (such as time slots, frequency bands, or codewords). Therefore, within a single cell, users do not interfere with each other. In this case, the signal-to-interference-plus-noise ratio (SINR) for user m is calculated using the following formula: ; Where P m h is the transmit power allocated by the serving base station to user m. m This is the channel gain between user m and the serving base station. The summation term in the denominator... This represents the sum of small interferences from all other neighboring cell base stations. Among them, It is the transmission power that the interfering base station allocates to user m. σ² represents the channel gain between user m and the interfering base station. σ² represents the power of additive white Gaussian noise.

[0034] This embodiment quantifies the overall user experience quality (QoE) as system service satisfaction (S), specifically defined as the proportion of users whose communication needs are met out of the total number of users. This indicator considers both the throughput requirements of eMBB services and the reliability and latency requirements of URLLC services. Its mathematical expression is as follows: ; Among them, M eMBB and M URLLC The total number of users in the two categories, respectively; and For binary indicator variables: when the throughput requirements of eMBB user m are met, Otherwise, it is 0; when the latency and reliability requirements of URLLC user m are simultaneously met, Otherwise, it is 0; the specific service quality requirements are given by the following formula: ; in, r e , ε uand t u These represent the actual throughput of eMBB users and the actual transmission error rate and latency of URLLC users, respectively. r s e , ε s u and t s u This corresponds to the service quality threshold, namely the minimum rate required for eMBB, the maximum error rate allowed by URLLC, and the maximum latency.

[0035] The resource allocation optimization problem for combining eMBB and URLLC services is expressed as: ; ; ; ; Among them, P m P is the transmit power allocated by the serving base station to user m; max η is the maximum transmit power of the system. m This indicates the time-domain preemption ratio of URLLC users over eMBB users. x k m This indicates whether resource block k has been allocated to user m.

[0036] 2. Markov Decision Process Modeling: In the MDP model, the agent acts as the decision-making body, perceives the network environment by observing the state, and performs actions accordingly to implement resource allocation. The environment then transitions to a new state based on the actions performed and provides a reward signal to evaluate the immediate benefits of the actions. Through a closed-loop interaction consisting of state, action, and reward, the intelligent agent gradually learns a dynamic resource allocation strategy that maximizes long-term cumulative returns.

[0037] The intelligent agent is defined as follows: It acts as the central controller of a single base station, responsible for making unified resource allocation decisions within its coverage area for wireless network scenarios involving both eMBB and URLLC service users. This intelligent agent dynamically optimizes and allocates spectrum resources by observing the global network status and outputting continuous control actions.

[0038] The state space is defined as: the state of the system at each moment. S tIt is a high-dimensional vector that comprehensively represents the real-time operating status of the network, specifically including the information-to-dryness ratio γ for each user. m and data type identifier for each user Type m ( Type m =1 indicates the URLLC type. Type m = 0 indicates eMBB type). The mathematical expression of the state vector is: S t = {γ m , Type m}

[0039] The action space is defined as: the time slot in which each agent operates. t The action set consists of key decision variables in the resource allocation optimization problem of joint eMBB and URLLC services, and is defined as a three-dimensional continuous action vector. a ( t )= { α rb , β pw , η}. In the action set α rb The quantity represents the adjustment coefficient for allocating resource blocks based on an average allocation. β pw This represents the adjustment factor for the transmission power based on the average distribution. η This represents the proportion of time slot resources that URLLC services seize from eMBB services. First, an average allocation baseline for resource blocks and power is calculated based on the number of users, and then fine-tuning is achieved through adjustments in the action vector. α rb and β pw The definition range is between 0 and 2, and the micro-slot preemption ratio is... η In actual execution, it is quantized into discrete values ​​[0, 2 / 14, 4 / 14, 7 / 14, 1.0] to match the 5G frame structure.

[0040] The reward function is defined as follows: Design a multi-objective weighted reward function. R ( t ): R ( t ) = w urllc × R urllc +w eMBB × ReMBB + w efficiency × R efficiency ; in, R ( t This represents an immediate reward at time step t, designed to simultaneously optimize the differentiated quality of service requirements of eMBB and URLLC services. w urllc , w eMBB , w efficiency All are weighting coefficients.

[0041] The reward formula for URLLC services is: ; ; ; Where, ε s ε is the system's tolerance threshold for error rate. u For real-time error rate, t s t is the system's tolerance time delay threshold. u For real-time latency, w error As the weight of the error rate, w delay The weights are used to account for latency; when the service requirement threshold is exceeded, a logarithmic function is used to avoid training instability caused by extreme values.

[0042] eMBB service reward formula: ; in, This indicates the percentage of eMBB users who meet the service requirements. To meet the eMBB rate requirements of users, M eMBB This represents the total number of eMBB users. This incentive aims to improve overall service satisfaction for eMBB slices by maximizing the proportion of eMBB users meeting rate requirements, thereby promoting fairness in resource allocation among users.

[0043] R efficiency To evaluate preemption efficiency based on the Spearman rank correlation coefficient, and to encourage users to preempt time slot resources from those with ample frequency domain resources or good channel quality, it is expressed as: ; ; ; in, ρ 1. ρ 2 is the rank correlation coefficient. ρ 1. Calculate the correlation between eMBB user resource allocation and preemption ratio. ρ 2. Calculate the correlation between signal-to-noise ratio and preemption ratio.

[0044] Weighting coefficient w urllc , w eMBB , w efficiency , w error and w delay The method was determined through experimental optimization and is used to balance the relative importance of different business objectives.

[0045] r ratio The ratio used to measure the actual rate of return to the demand rate is expressed as: ; ; ; in, r actual It is the actual rate value calculated based on resource blocks, power, and micro-slot preemption. r require The required rate is calculated based on the current data volume, where η is the time slot preemption ratio. x k m This indicates whether resource block k has been allocated to user m. f k The bandwidth of a resource block. γ m Let D be the signal-to-interference-plus-noise ratio (SIR) for user m, D be the channel dispersion, and c be the block length. v For error rate, B u To serve the amount of data transmitted by users, τ Duration of URLLC service.

[0046] R guidance To measure whether the average micro-slot preemption ratio is reasonable, it is set as a piecewise function, expressed as: ; in, W guidance To maximize the weight of the guidance and reward portions, penalties are imposed when the allocation is insufficient or excessive, while rewards are given when the demand is just met. rratio It is the ratio of the actual rate to the required rate.

[0047] To enhance the agent's ability to perceive the long-term impact of resource allocation strategies, this invention will R guidance The network is constructed as multi-scale temporal features. Specifically, this invention designs a lightweight feature extraction network, which takes short-term (10 steps), medium-term (100 steps), and long-term (200 steps) data as inputs. R guidance Historical sequence. The network fuses features from different time scales into a 16-dimensional feature embedding vector, which is then used as additional state input to the Actor network.

[0048] Unlike studies that widely employ Long Short-Term Memory (LSTM) networks to process temporal information, this invention chooses a customized lightweight feedforward network for feature fusion, primarily based on the following two points: First, although LSTM can automatically learn temporal dependencies, its complex gating structure introduces more computational overhead and parameters, which is detrimental to the policy network's rapid response during inference, potentially conflicting with the low-latency decision-making requirements of URLLC services. Second, for specific trends that need attention in this scenario (such as short-term fluctuations and long-term equilibrium in preemption ratios), multi-scale features are explicitly constructed through fixed time windows, providing the agent with more interpretable and targeted guidance, avoiding the noise and convergence uncertainty that may arise from the implicit learning of LSTM from the original sequence. This design, while ensuring temporal awareness capabilities, achieves a simpler network structure, faster training speed, and ultimately efficiently assists the agent in achieving more accurate dynamic resource allocation.

[0049] 3. Algorithm flow based on slice-aware dual-delay deep deterministic policy gradient The slice-aware, dual-delay deep deterministic policy gradient algorithm (SA-TD3) is based on the TD3 framework. By introducing mechanisms such as dual Critic networks, target policy smoothing regularization, and delayed policy updates, it effectively alleviates the Q-value overestimation problem common in traditional algorithms, improving the algorithm's stability and convergence. Through its unique state, action, and reward design, it achieves deep perception and adaptive optimization of network slice scenarios. Specifically, the slice-aware, dual-delay deep deterministic policy gradient algorithm SA-TD3 adopts an Actor-Critic architecture, the core framework of which is as follows: Figure 2 As shown: The intelligent agent (base station) observes the network status. S t The extracted preemptive reward features are used to output continuous actions via the Actor network. a t This includes resource blocks, power, and preemption ratio; two independent Critic networks are used to evaluate the long-term value of this state-action pair.Q 1( S t , a t ), Q 2( S t , a t The smaller value is taken as the target Q value to reduce overestimation bias and guide the policy update of the Actor network. At the same time, an experience replay buffer is created to store the interaction data between the agent and the environment. The introduction of the experience replay mechanism ensures the stability of the learning process.

[0050] The training process of SA-TD3 is as follows: First, the Actor network is randomly initialized. With two Critic networks , The parameters, These are the parameters of the neural network. These parameters are then copied to their respective target networks. , and To provide a stable learning objective, an experience replay buffer R of size N is created to store the agent's interaction data with the environment, and a stochastic process N (typically Gaussian noise) is initialized to encourage exploration early in training. Key hyperparameters are also set, including the policy update frequency d, the variance σ of the target policy's smoothing noise, and the action clipping range. And the soft update coefficient τ, etc.

[0051] Training is conducted in multiple episodes. In each episode, the agent (i.e., the central controller of the base station) first observes the current network state. This state is a composite vector containing the information-to-dryness ratio for each user. and business type identifier (1 represents URLLC, 0 represents eMBB). Next, the Actor network outputs a basic 3D continuous action vector based on this state. .in, It is the adjustment factor for resource block allocation. These are the adjustment coefficients for transmit power, and both range from 0 to 2. η represents the proportion of time slots that URLLC services preempt from eMBB services, which is quantized into discrete values ​​{0, 2 / 14, 4 / 14, 7 / 14, 1.0} to match the 5G frame structure during actual execution. To facilitate exploration, noise will be added to the basic operations. And crop the final result: The base station then performs this action, allocating resources according to the adjusted plan. The environment then comprehensively assesses the impact of this resource allocation on multiple objectives, including eMBB throughput, URLLC latency, and reliability, and calculates and feeds back an immediate scalar reward. This reward is the core guiding signal for algorithm learning; it quantifies the immediate effect of the current action on meeting the quality of service requirements of heterogeneous slices. The environment then provides an immediate reward. and the next state The experience tuple of this interaction ( It is stored in the experience replay buffer.

[0052] Once the amount of data in the buffer reaches the preset batch size, the network update step begins. First, a batch of N experience samples is randomly sampled from the buffer. During the update process, rewards are... The target Q-value is a crucial input for calculating the objective Q-value, directly determining the baseline direction for state-action pair value estimation. The Critic network update is guided by calculating the objective Q-value: first, smoothing noise is added to the objective action, i.e. , The noise follows a Gaussian distribution; then, the smaller value of the two Critic target network outputs is taken to calculate the target. , where γ is the discount factor. Next, by minimizing the current Critic network Q-value estimate relative to the target... The mean squared error loss between the two networks is used to update the parameters of the Critic network. The Actor network, on the other hand, employs a delayed update strategy, executing once every d steps. Its update direction follows the policy gradient that maximizes the Q-value, which is the long-term value assessment learned by the Critic network based on historical rewards. Therefore, rewards indirectly guide the optimization direction of the Actor policy by influencing the Q-value. This gradient is jointly determined by the gradient of the Critic network with respect to actions and the gradient of the Actor network with respect to parameters. Finally, all target networks slowly track the parameters of their corresponding main networks using a soft update method. , , where τ is a coefficient much smaller than 1, ensuring the stability of the learning process.

[0053] The entire training process is repeated until the preset number of rounds is reached or performance convergence is achieved. Through the above steps, the SA-TD3 algorithm can autonomously learn a resource allocation strategy that can simultaneously guarantee high throughput of eMBB and low latency and high reliability of URLLC under time-varying channels and burst traffic by continuously interacting with the dynamic environment and guided by reward signals. Its dual-critic structure and delayed update mechanism effectively alleviate the problem of value function overestimation, thereby improving the stability and final performance of the algorithm.

[0054] The slice-aware feature of the SA-TD3 algorithm is not implemented through a separate module, but rather by deeply integrating slice information into the core mechanism of the algorithm. Specifically, in each decision step, the algorithm receives information including the user's service type. Type m status S t This enables the Actor network to perceive the existence of different slices from the ground up. Through a carefully designed reward function, the algorithm gradually learns the service quality requirements of each slice, forming an optimization objective. Based on this learning process, policy generation can adapt to the differentiated characteristics of services and achieve dynamic resource allocation. Furthermore, the experience replay mechanism utilizes historical data from interactions with users in different slices, allowing the algorithm to adapt to diverse business scenarios, including the burstiness of URLLC and the continuous demands of eMBB. This deeply integrated design makes SA-TD3 a context-aware dynamic resource management strategy, thereby achieving efficient coexistence of eMBB and URLLC services in a shared resource environment.

[0055] 4. Simulation Experiment 4.1 Experimental Environment and Parameter Configuration The channel model considered in this invention takes into account path loss, shadowing fading, and Ricean fading, and their mathematical representations are as follows. Path loss PL can be expressed as: ; in, d ( t This indicates the distance between the base station (BS) and the user. f c Represents the carrier frequency. Shadow fading follows a log-normal distribution with a standard deviation of 4 dB. Small-scale fading is described by Ricean fading, which includes line-of-sight (LOS) and non-line-of-sight (NLoS), as shown in the following formula: ; The line-of-sight distance (LoS) represents the direct component. Doppler frequency shift v represents the user's speed, and c represents the speed of light. θ This represents the angle between the user's movement direction and the LOS path. φ This is the initial phase. The non-line-of-sight (NLOS) component represents the scattering component, which follows a Rayleigh distribution. The non-line-of-sight (NLoS) component is generated using the Jakes model. S represents the number of sine waves. Indicates amplitude. Indicates Doppler frequency shift, The phase shift is represented and normalized to [0, 2π]. Table 1 lists the parameters used in the simulation.

[0056] Table 1 Simulation Parameters 4.2 Experimental Results 4.2.1 Algorithm Training Convergence and Performance Analysis The proposed SA-TD3 algorithm was initially configured with 20 eMBB users for basic training. The training process is as follows: Figure 3 As shown, in the early stage of training, the cumulative reward fluctuates significantly due to the high exploration rate; as training progresses, the strategy gradually stabilizes and converges after about 1000 training cycles, indicating that the algorithm has a fast convergence speed and good stability.

[0057] Under the convergence strategy, the algorithm demonstrates excellent service assurance capabilities across key metrics such as eMBB average rate, URLLC latency, and reliability. The eMBB average service rate is as follows: Figure 4 As shown, the average throughput of the eMBB service remained stable at approximately 6.08 Mbps, higher than the set service threshold of 5 Mbps. The transmission latency of the URLLC service is as follows... Figure 5 As shown, in all test batches, the transmission latency of the URLLC service was strictly below the threshold of 0.5ms, indicating that the algorithm can effectively guarantee the real-time requirements of low-latency services. Figure 6 The cumulative probability distribution of URLLC transmission error rate is shown, and it can be seen that the test results are generally below 10⁻. 4 The target threshold was used to verify the effectiveness of the algorithm in ensuring high reliability.

[0058] To verify the adaptability of the proposed algorithm under different business scenarios, this invention conducted a series of comparative experiments, examining the impact of different numbers of eMBB users (M=20, 25, 30, 35, 40, a total of 5 scenarios) and different URLLC data packet sizes (L=0, 256, 512, 1024 bits, a total of 4 scenarios) on system performance.

[0059] Experimental results show that, under different configurations, the total user speed of the system remains relatively stable at around 145 Mbps. Figure 7 As shown; the spectrum utilization rate remains at approximately 8 bit / s / Hz, as Figure 8 As shown. The eMBB service speeds for different services are compared, for example... Figure 9 As shown, as the number of eMBB users increases, the average rate of a single user gradually decreases, but the total user rate of the system remains stable, indicating that the algorithm has good scalability. Figure 10 The presentation showcases URLLC transmission latency under different business scenarios. Latency in all scenarios is less than 1ms, meeting real-time requirements. Furthermore, the trend of URLLC transmission error rate with data packet size is shown below. Figure 11As shown, the error rate increases with the increase in data packet size, but it still remains basically at 10%. -3 This indicates that the algorithm has reliable performance. The algorithm demonstrates good robustness under dynamic scenarios such as different URLLC packet arrival rates and the number of eMBB users, further validating its adaptability and application potential in real-world environments.

[0060] 4.2.2 Algorithm Comparison and Analysis To comprehensively evaluate the performance of the proposed algorithm, it is compared with the following three benchmark methods.

[0061] (1) Static allocation strategy: a fixed ratio is used, with 25% of the resources reserved for URLLC users and 75% of the resources reserved for eMBB users, and the power is evenly distributed among users.

[0062] (2) The SA-DQN allocation strategy has the same observation space and reward space as SA-TD3, except that the resource blocks and power adjustment ratios in the action space are changed to 5 selectable actions, such as / =[0.5, 0.8, 1.0, 1.2, 1.5] times the average resource distribution.

[0063] (3) Compare with the O-RAN-based Distributed Multi-Agent DRL Algorithm.

[0064] This invention uses charts to compare and analyze the algorithm performance from four dimensions: spectrum utilization, average eMBB rate, eMBB service fairness index, and URLLC service error rate. Spectrum utilization comparison... Figure 12 As shown, the SA-TD3 algorithm outperforms static allocation and O-RAN-DDRL in terms of overall system spectral efficiency. The average eMBB rate is compared to... Figure 13 As shown, SA-TD3's average rate is close to that of O-RAN-DRRL and higher than other algorithms. The eMBB service fairness index is comparable to... Figure 14 As shown, the fairness index of SA-TD3 is higher than that of O-RAN-DDRL, indicating that resource allocation among users is more balanced. The URLLC service transmission error rate is compared to... Figure 15 As shown, the error rate of SA-TD3 is lower than that of O-RAN-DDRL, SA-DQN, and static allocation.

[0065] To evaluate the overall performance of the algorithms, Table 2 summarizes the average values ​​of each comparison algorithm on key performance indicators. To quantitatively assess overall performance, a comprehensive scoring system was used, integrating all four indicators through a weighted normalization process. The weights were allocated as follows: spectral efficiency (25%), eMBB average rate (30%), eMBB fairness index (20%), and URLLC transmission error probability (25%). Each indicator was normalized to the range [0, 1]: for positive indicators (higher values ​​are better), the normalized score was calculated as ((value - minimum value)) ÷ ((maximum value - minimum value)); for negative indicators (URLLC error probability, lower values ​​are better), the normalized score was calculated as ((maximum value - value)) ÷ ((maximum value - minimum value)). The overall score was then obtained by summing the weighted normalized scores.

[0066] Based on this scoring method, the proposed SA-TD3 algorithm achieved the highest overall score of 88.05, indicating that it achieves an optimal balance between spectral efficiency (8.0612 bps / Hz), average eMBB service rate (6.09 Mbps), user fairness (0.9452), and URLLC service transmission error probability (0.000167). In contrast, the SA-DQN algorithm scored 85.40, performing better in spectral efficiency (8.0912 bps / Hz) and fairness (0.9670), but its average eMBB service rate (5.95 Mbps) was lower, and its URLLC error rate (0.000625) was higher than SA-TD3. The O-RAN-DDRL algorithm scored 73.47, providing an acceptable average eMBB rate (6.23 Mbps), but its user fairness index (0.8228) was significantly lower, indicating deficiencies in resource allocation balance. The baseline static allocation strategy scores only 20.00; although it is extremely fair (0.9987), it has the lowest spectral efficiency and eMBB service rate, and poor URLLC error performance, so it is not suitable for dynamic service scenarios.

[0067] Table 2 Average performance statistics of each algorithm The above comparison results show that, for the task of dynamic resource allocation in multi-service coexistence scenarios, under the same experimental settings and evaluation standards, SA-TD3 demonstrates outstanding performance in key performance indicators such as overall system efficiency, user fairness, and URLLC service reliability, showing superior comprehensive performance and practical value, and verifying its effectiveness and competitiveness in solving this problem.

[0068] By modeling differentiated business requirements as a Markov decision process and designing a slice-aware state, action, and reward mechanism within the TD3 framework, the algorithm can adaptively coordinate micro-slot preemption, resource block allocation, and power control to achieve an effective balance between multiple service quality objectives. Simulation results show that the SA-TD3 algorithm has good convergence and training efficiency, converging quickly within approximately 1000 training cycles; it also ensures eMBB service rates (averaging 6.09 Mbps) and URLLC service low latency (<0.5ms) and high reliability (error rate less than 10⁻⁻⁶). 4 It performs excellently in all aspects; in terms of system spectrum utilization (reaching 8.06 bps / Hz), it is significantly better than the static allocation (7.41 bps / Hz) method, demonstrating higher intelligence in resource allocation.

[0069] The SA-TD3 algorithm of this invention demonstrates excellent performance across multiple metrics, providing a feasible intelligent resource allocation solution for the efficient coexistence of heterogeneous services in 5G-Advanced and future 6G networks. It possesses significant theoretical research value and practical application potential. Future development will integrate offline learning and online fine-tuning to further enhance its generalization ability and deployment flexibility.

[0070] This invention first initializes network parameters and a learning model. Then, the base station collects user signal-to-dryness ratios and service types in each transmission interval, forming a state vector. Next, the agent outputs resource block adjustment coefficients, power adjustment coefficients, and micro-slot preemption ratios based on the current state, and the base station performs real-time resource allocation accordingly. After resource allocation, a reward signal is calculated based on eMBB user rate satisfaction and URLLC user latency and reliability performance, and this interaction experience is stored in an experience replay buffer. In scenarios supporting online learning, the system periodically samples experience data to update the agent's network parameters to continuously optimize decision-making strategies. Finally, the system state is updated, and the system enters the scheduling loop for the next time slot. Through this closed-loop process, this method achieves adaptive allocation of multi-service resources in a dynamic network environment, balancing spectrum efficiency, user fairness, and differentiated quality of service assurance.

[0071] The above-described specific embodiments are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the above-described specific embodiments. Any appropriate changes or modifications made by a person skilled in the art that conform to the claims of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A slice-aware dynamic resource allocation method based on deep reinforcement learning, characterized in that, Specifically, the following steps are included: Step 1: Identify the resource allocation optimization issues for the combined eMBB and URLLC services; Step 2: Construct the resource allocation optimization problem in Step 1 into a Markov Decision Process (MDP) model, and identify its core elements, which include the agent, state space, action space, and reward function. Step 3: Design a dual-delay deep deterministic strategy gradient algorithm for slice awareness. The algorithm relies on the TD3 framework to jointly optimize time and frequency resources and transmit power, realize dynamic adaptation to network slice requirements, and achieve adaptive allocation of multi-service resources in dynamic network environments.

2. The slice-aware dynamic resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, Step 1 includes the following steps: The overall user experience quality (QoE) is quantified as system service satisfaction (S), specifically defined as the proportion of users whose communication needs are met out of the total number of users. This indicator considers both the throughput requirements of eMBB services and the reliability and latency requirements of URLLC services. Its mathematical expression is as follows: ； Among them, M eMBB and M URLLC The total number of users in the two categories, respectively; and For binary indicator variables: when the throughput requirements of eMBB user m are met, Otherwise, it is 0; when the latency and reliability requirements of URLLC user m are simultaneously met, Otherwise, it is 0; the specific service quality requirements are given by the following formula: ； in, r e , ε u and t u These represent the actual throughput of eMBB users and the actual transmission error rate and latency of URLLC users, respectively. r s e , ε s u and t s u Then it corresponds to the service quality threshold, namely the minimum rate required by eMBB, the maximum error rate allowed by URLLC, and the maximum latency. The resource allocation optimization problem for combining eMBB and URLLC services is expressed as: ； ； ； ； Among them, P m P is the transmit power allocated by the serving base station to user m; max η is the maximum transmit power of the system. m This indicates the time-domain preemption ratio of URLLC users over eMBB users. x k m Indicates whether resource block k has been allocated to user m: ； 。 3. The slice-aware dynamic resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, Step 2 includes: In the MDP model, the agent acts as the decision-making body, perceives the network environment by observing the state, and performs actions accordingly to implement resource allocation. The environment then transitions to a new state based on the actions performed and provides a reward signal to evaluate the immediate benefits of the actions. Through a closed-loop interaction consisting of state, action, and reward, the intelligent agent gradually learns a dynamic resource allocation strategy that maximizes long-term cumulative returns.

4. The slice-aware dynamic resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In step 2, the intelligent agent is defined as follows: the intelligent agent is the central controller of a single base station, responsible for making unified resource allocation decisions for wireless network scenarios that include both eMBB and URLLC service users within its coverage area.

5. The slice-aware dynamic resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In step 2, the state space is defined as: the state of the system at each time step. S t It is a high-dimensional vector that comprehensively represents the real-time operating status of the network, specifically including the information-to-dryness ratio γ for each user. m and data type identifier for each user Type m The mathematical expression of the state vector is: S t = {γ m , Type m } 6. The slice-aware dynamic resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In step 2, the action space is defined as: the time slot in which each agent... t The action set consists of key decision variables in the resource allocation optimization problem of joint eMBB and URLLC services, and is defined as a three-dimensional continuous action vector. a ( t )= { α rb , β pw , η }; In the action set α rb The quantity represents the adjustment coefficient for allocating resource blocks based on an average allocation. β pw This represents the adjustment factor for the transmission power based on the average distribution. η This indicates the proportion of time slot resources that URLLC services seize from eMBB services.

7. The slice-aware dynamic resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In step 2, the reward function is defined as: designing a multi-objective weighted reward function. R ( t ): R ( t ) = w urllc × R urllc +w eMBB × R eMBB + w efficiency × R efficiency ； in, R ( t () represents the immediate reward at time step t; w urllc , w eMBB , w efficiency All are weighting coefficients; The reward formula for URLLC services is: ； ； ； Where, ε s ε is the system's tolerance threshold for error rate. u For real-time error rate, t s t is the system's tolerance time delay threshold. u For real-time latency, w error As the weight of the error rate, w delay Weights for latency; eMBB service reward formula: ； in, This indicates the percentage of eMBB users who meet the service requirements. To meet the eMBB rate requirements of users, M eMBB This represents the total number of eMBB users; R efficiency To evaluate preemption efficiency based on Spearman's rank correlation coefficient, it is expressed as: ； ； ； in, ρ 1. ρ 2 is the rank correlation coefficient. ρ 1. Calculate the correlation between eMBB user resource allocation and preemption ratio. ρ 2. Calculate the correlation between signal-to-noise ratio and preemption ratio; r ratio The ratio used to measure the actual rate of return to the demand rate is expressed as: ； ； ； in, r actual It is the actual rate value calculated based on resource blocks, power, and micro-slot preemption. r require The required rate is calculated based on the current data volume, where η is the time slot preemption ratio. x k m This indicates whether resource block k has been allocated to user m. f k The bandwidth of a resource block. γ m Let D be the signal-to-interference-plus-noise ratio (SIR) for user m, D be the channel dispersion, and c be the block length. v For error rate, B u To serve the amount of data transmitted by users, τ Duration of URLLC service; R guidance To measure whether the average micro-slot preemption ratio is reasonable, it is set as a piecewise function, expressed as: ； in, W guidance To maximize the weight of the guidance and reward portions, penalties are imposed when the allocation is insufficient or excessive, while rewards are given when the demand is just met. r ratio It is the ratio of the actual rate to the required rate.

8. The slice-aware dynamic resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, The slice-aware dual-delay deep deterministic policy gradient algorithm SA-TD3 adopts an Actor-Critic architecture. Its core framework is as follows: the agent observes the network state St and the extracted preemption reward features, and outputs continuous actions at via the Actor network, including resource blocks, power, and preemption ratio; two independent Critic networks are used to evaluate the long-term value Q1(St, at) and Q2(St, at) of the state-action pair, and take the smaller value as the target Q value to guide the policy update of the Actor network. At the same time, an experience replay buffer is created to store the interaction data between the agent and the environment.

9. The slice-aware dynamic resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, The training process of SA-TD3 is as follows: First, the Actor network is randomly initialized. With two Critic networks , The parameters, These are the parameters of the neural network; subsequently, these parameters are copied to their respective target networks. , and Simultaneously, an experience replay buffer R of capacity N is created to store the interaction data between the agent and the environment, and a random process N is initialized to encourage exploration in the early stages of training.

10. The slice-aware dynamic resource allocation method based on deep reinforcement learning according to claim 9, characterized in that, Once the amount of data in the buffer reaches the preset batch size, the network update process begins, which includes: First, randomly sample a batch of N experience samples from the buffer; during the update process, the reward... r t It is the key input for calculating the target Q value; The Critic network's updates are guided by calculating the target's Q-value: first, smooth noise is added to the target action, i.e. , The noise follows a Gaussian distribution; then, the smaller value of the two Critic target network outputs is taken to calculate the target. Where γ is the discount factor; then, by minimizing the Q-value estimate of the current Critic network relative to the target... The mean squared error loss between the two is used to update the parameters of the Critic network.

Images