5G Network Slicing Resource Allocation Method Based on Differentiated Service Time Scale
By using the MATD3 algorithm to perform coarse-grained and fine-grained resource allocation in 5G networks, the problem of resource allocation instability in dynamic heterogeneous environments of 5G network slicing is solved, achieving efficient and stable differentiated services and improving resource utilization and service quality.
Patent Information
- Application Number
- CN202512028189.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-11-17
AI Technical Summary
Existing 5G network slicing technology struggles to achieve efficient and stable resource allocation across different time scales in dynamic heterogeneous environments, resulting in low resource utilization and increased interference between slices. This fails to meet the stringent QoS requirements of URLLC (ultra-low latency), eMBB (high throughput), and mMTC (massive machine-type communication).
The Multi-Agent Dual-Delay Deep Deterministic Policy Gradient Algorithm (MATD3) is adopted to perform coarse-grained resource pre-allocation and business classification on a minute-level time scale, combined with fine-grained resource adjustment on a second-level time scale. Through centralized training and parameter distribution of the Actor network and dual Critic network, fine-grained resource adjustment and cross-slice priority preemption are achieved.
It improves the robustness and convergence efficiency of multi-agent collaboration, optimizes network resource utilization and service quality, and ensures that differentiated QoS for various services is met.
Smart Images

Figure CN123124114A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of communication technology, and in particular relates to a 5G network slicing resource allocation method based on differentiated service time scales. Background Technology
[0002] With the deepening of 5G commercial deployment, network slicing technology has become a core means to meet the differentiated needs of three typical scenarios: eMBB, URLLC, and mMTC. Through NFV / SDN and SBA service architecture, the physical network can be divided into multiple logically independent virtual networks, realizing "one network for multiple uses," end-to-end resource isolation, and customized services.
[0003] However, in real-world, highly dynamic, heterogeneous environments, existing multi-timescale slice resource allocation technologies still have significant shortcomings: traditional solutions, while able to guarantee basic slice isolation through minute-level coarse-grained pre-allocation, struggle to adapt to second-level or even millisecond-level drastic user movement, channel fluctuations, and sudden URLLC tasks, leading to low resource utilization and increased inter-slice interference; existing single-critically-multi-agent reinforcement learning methods based on MADDPG and other methods are prone to Q-value overestimation, policy oscillations, and slow convergence speeds in some observable scenarios, causing frequent fluctuations in resource allocation decisions and making it difficult to simultaneously meet the stringent QoS requirements of ultra-low latency for URLLC, high throughput for eMBB, and massive access for mMTC; furthermore, existing technologies generally lack effective coarse- and fine-grained timescale coordination mechanisms and cross-slice priority preemption strategies, resulting in a sharp decline in service quality and severe overall resource waste during traffic bursts.
[0004] These shortcomings make it difficult for existing 5G network slicing to achieve efficient and stable resource allocation in complex and dynamic scenarios, limiting its large-scale application in high-reliability scenarios such as industrial internet, vehicle-to-everything (V2X) and telemedicine. Summary of the Invention
[0005] The purpose of this invention is to provide a 5G network slice resource allocation method based on differentiated service time scales, so as to solve the problem that the existing technology cannot achieve efficient and stable allocation of multiple slice resources under differentiated time scales in the dynamic heterogeneous environment of 5G networks, while improving the robustness and convergence efficiency of multi-agent cooperation.
[0006] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides a 5G network slice resource allocation method based on a differentiated service time scale, the method comprising:
[0007] Within a minute-level timescale, each base station classifies services based on the service demand vectors reported by the terminals in real time, and performs coarse-grained resource pre-allocation and constructs a virtual slice framework based on the classification results;
[0008] A multi-agent dual-delay deep deterministic policy gradient algorithm is adopted. Based on the interaction experience data continuously uploaded by each base station, the Actor network and dual Critic network corresponding to all base stations are trained in a global priority experience replay pool. After the training converges or the update cycle is reached, the latest Actor network parameters and dual Critic network parameters are sent to each base station.
[0009] Within a second-level timescale, the latest Actor network parameters and dual Critic network parameters are combined with real-time collected local state observation information to perform deterministic policy reasoning, obtain fine-grained resource adjustment actions, and execute them.
[0010] As a further improvement to one embodiment of the present invention, the method further includes, in that, classifying the services by each base station according to the service demand vector reported by the terminal in real time, the following steps are included:
[0011] The business requirement vector is defined as follows: ;in, For the number of bits in the task, The number of CPU cycles required per bit. For the number of user connections;
[0012] Each base station performs service classification based on the aforementioned service demand vector, including:
[0013] when When the value exceeds a first preset threshold, the task is classified into an eMBB slice; where, Maximum number of bits for the task;
[0014] when When the value exceeds the second preset threshold, the task will be classified into a URLLC slice; where, The maximum number of CPU cycles required per bit;
[0015] when When the value exceeds the third preset threshold, the task will be classified into an mMTC slice; among which, This represents the maximum number of user connections.
[0016] As a further improvement to one embodiment of the present invention, the method further includes, in which the step of pre-allocating coarse-grained resources based on the classification results and constructing a virtual slice framework includes,
[0017] The coarse-grained resource pre-allocation obtains the basic resource quantities for various slices by solving a resource pre-allocation optimization problem that incorporates proportional fairness and exponential utility. The formula is:
[0018]
[0019]
[0020] in, , Given the spectrum allocated to slice s and the total amount of computing resources, For the historical average demand of slice s, The priority weights of slice s This is the balance coefficient between the proportional equity term and the index utility term. Let be the resource sensitivity coefficient of slice s. This refers to the total spectrum and computing resource capacity of the base station. The minimum guaranteed resource for slice s;
[0021] According to the solution We use hard slicing and / or soft slicing technologies to build independent virtual network slicing frameworks for the three types of slices and complete the initial resource binding.
[0022] As a further improvement to one embodiment of the present invention, the method further includes, in the step of centrally training the Actor network and dual Critic network corresponding to all base stations in the global priority experience replay pool, the following steps are included:
[0023] The local observation information uploaded by each base station, the resource allocation actions performed, the rewards obtained, and the local observation information at the next moment are continuously collected as interactive experience data and stored in the global priority experience replay pool.
[0024] The local observation information of each base station n is defined as follows: ;in, This represents the local observation information of base station n at time t. These are the location, velocity, channel state, and service demand vector of user i, respectively. The set of users within the service range of base station n;
[0025] Construct the global state based on the local observation information of all base stations. and the coordinated actions of all base stations. .
[0026] As a further improvement to one embodiment of the present invention, the method further includes, the updating of the dual-Critic network includes,
[0027] Add truncated Gaussian noise to the target Actor output to generate exploratory target actions. The formula is:
[0028] ;
[0029] in, Represents the target Actor network. This represents the global state at the next moment. For the target Actor network parameters, It is Gaussian noise. For noise standards, The truncation range;
[0030] Based on the exploratory target action, the dual-Critic network calculates the target Q-value. The formula is:
[0031] ;
[0032] in, This represents the immediate reward received by base station n. As a discount factor, Describes the critic network for the i-th target. Let i be the network parameters for the i-th target Critic. For each base station, explore joint target actions;
[0033] The two Critic networks are synchronized by minimizing the temporal difference loss, as shown in the formula:
[0034] ;
[0035] in, Let r be the expected value of the Critic loss function, and r be the reward obtained. This refers to the online Q-value.
[0036] As a further improvement to one embodiment of the present invention, the method further includes updating the Actor network, including...
[0037] After the Critic network fully converges, the Actor network is updated once every preset number of steps, and the policy gradient is expressed as:
[0038] ;
[0039] in, Let the policy objective function be... Let be the expected gradient of the Actor's policy. As the first online Critic network, For the Actor network of base station n, Let n be the Actor network parameters for base station n;
[0040] The target Critic network is updated using a soft update mechanism, with the following formula:
[0041] ;
[0042] ;
[0043] in, This is the soft update coefficient.
[0044] As a further improvement to one embodiment of the present invention, the method further includes performing deterministic policy reasoning on the latest Actor network parameters and real-time collected local state observation information to obtain fine-grained resource adjustment actions and executing them.
[0045] Using the issued Actor parameters Real-time local observation Deterministic policy reasoning yields the following fine-grained action representation:
[0046] ;
[0047] in, This represents the fine-grained resource adjustment action output by base station n at time t, including the spectrum allocation ratio for each user i. and computing resource allocation ratio ;
[0048] Will The mapping is to specific resource operations, including micro-slot pre-occupancy scheduling for URLLC slice users, carrier aggregation scheduling for eMBB slice users, and narrowband multiplexing scheduling for mMTC slice users.
[0049] When a momentary shortage of URLLC slice resources is detected, resources are preferentially borrowed from the free resource blocks of eMBB or mMTC slices, and the URLLC slice is increased in the next minute-level cycle. Compensation will be provided.
[0050] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a 5G network slice resource allocation system based on differentiated service time scales, the system comprising a slice management module, a centralized training module, and a decision execution module;
[0051] The slice management module is used to classify the services of each base station according to the service demand vector reported by the terminal in real time within a minute-level time scale, and to pre-allocate coarse-grained resources and construct a virtual slice framework based on the classification results.
[0052] The centralized training module is used to employ a multi-agent dual-delay deep deterministic policy gradient algorithm. Based on the interaction experience data continuously uploaded by each base station, it centrally trains the Actor network and dual Critic network corresponding to all base stations in the global priority experience replay pool. After the training converges or reaches the update cycle, the latest Actor network parameters and dual Critic network parameters are sent to each base station.
[0053] The decision execution module is used to perform deterministic policy reasoning on the latest Actor network parameters and dual Critic network parameters and real-time collected local state observation information within a second-level time scale, so as to obtain fine-grained resource adjustment actions and execute them.
[0054] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the 5G network slicing resource allocation method based on the differentiated service time scale as described above.
[0055] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps in the 5G network slicing resource allocation method based on the differentiated service time scale described above.
[0056] Compared with existing technologies, this invention provides a 5G network slicing resource allocation method based on differentiated service time scales. By introducing a hierarchical mechanism of minute-level coarse-grained resource pre-allocation and second-level fine-grained dynamic adjustment based on the MATD3 algorithm, it effectively solves the problems of unstable resource allocation and insufficient real-time performance in dynamic environments in existing technologies. At the same time, by adopting dual-critic minimum Q-value selection, target action noise perturbation, and delay policy updates, it suppresses Q-value overestimation and policy oscillation, improves the convergence efficiency and robustness of multi-agent cooperation, and thus achieves overall optimization of network resource utilization and service quality while ensuring differentiated QoS for various services. Attached Figure Description
[0057] Figure 1 This is an overall flowchart of the 5G network slicing resource allocation method based on differentiated service time scales as described in this invention.
[0058] Figure 2 This is a schematic diagram of the 5G network layered architecture of the 5G network slicing resource allocation method based on differentiated service time scales as described in this invention.
[0059] Figure 3This is a diagram of the MATD3 algorithm structure of the 5G network slicing resource allocation method based on differentiated service time scales as described in this invention.
[0060] Figure 4 This is a flowchart of the training process for the MATD3-based resource allocation algorithm of the 5G network slicing resource allocation method based on differentiated service time scale described in this invention.
[0061] Figure 5 This is a flowchart of the 5G network slice generation and resource allocation process of the 5G network slice resource allocation method based on differentiated service time scale described in this invention.
[0062] Figure 6 This is a schematic diagram of the architecture of the 5G network slicing resource allocation system based on differentiated service time scales as described in this invention. Detailed Implementation
[0063] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.
[0064] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0065] In Embodiment 1 of the present invention, the present invention provides a 5G network slice resource allocation method based on a differentiated service time scale, such as... Figure 1 As shown, the method includes,
[0066] S1: Within a minute-level timescale, each base station classifies services based on the service demand vectors reported by the terminals in real time, and performs coarse-grained resource pre-allocation and constructs a virtual slice framework based on the classification results;
[0067] S2: Employ a multi-agent dual-delay deep deterministic policy gradient algorithm. Based on the interaction experience data continuously uploaded by each base station, train the Actor network and dual Critic network corresponding to all base stations in a global priority experience replay pool. After the training converges or the update cycle is reached, send the latest Actor network parameters and dual Critic network parameters to each base station.
[0068] S3: Within a second-level timescale, the latest Actor network parameters and dual Critic network parameters are combined with real-time collected local state observation information to perform deterministic policy reasoning, obtain fine-grained resource adjustment actions, and execute them.
[0069] It should be noted that the present invention is based on, as follows Figure 2 The 5G network slicing layered framework shown is implemented. The 5G network slicing layered framework includes a terminal device layer, an edge processor layer, and a cloud server layer.
[0070] The terminal equipment layer is primarily responsible for reporting user service requirements in real time and receiving resource allocation instructions, specifically as follows: Service Requirement Reporting: User equipment reports its service requirements to the network in real time. These requirements include key information such as task type (different service types like eMBB, URLLC, mMTC), latency sensitivity (i.e., the service's tolerance for latency), and computational load (the amount of computing resources required to complete the task). This information is crucial for network resource allocation and slice management. Instruction Reception and Execution: The terminal equipment receives resource allocation instructions from the base station and executes corresponding operations according to the instructions to ensure that services operate normally on the allocated resources.
[0071] The edge processor layer deploys an SDN controller in each base station, containing minute-level and second-level processing modules for coarse-grained slice construction, resource pre-allocation, and fine-grained dynamic resource adjustment. Base stations share local state through SDN interfaces to achieve collaborative decision-making. SDN Controller Deployment: Each base station deploys an SDN controller integrating minute-level and second-level processing modules. The minute-level module handles coarse-grained slice construction and resource pre-allocation, while the second-level module handles fine-grained dynamic resource adjustment, thus enabling hierarchical management of network resources. Inter-Base Station Collaboration Mechanism: Base stations share local state through SDN interfaces. This sharing method does not rely on global information, reducing information exchange overhead and enabling collaborative decision-making among base stations to a certain extent, playing a role in scenarios such as interference coordination and load balancing.
[0072] The cloud server layer centrally trains the global policy model of the MATD3 algorithm through a central controller, aggregating local observation data, action decisions, and environmental feedback rewards (such as latency) uploaded by agents from each base station to construct a global experience replay pool. A distributed policy gradient algorithm is used to jointly optimize the Actor-Critic network parameters of all base station agents: the Actor network learns dynamic resource allocation policies by maximizing the Q-value of the Critic evaluation, while the Critic network uses global state-action pairs to calculate the expected long-term reward, guiding the policy update direction. During training, a target network soft update mechanism is adopted to progressively synchronize policy parameters to improve convergence stability. Priority sampling technology is also introduced to focus on learning high-value experiences (such as successful URLLC resource preemption cases). After training, the central controller distributes the updated model parameters to each base station through an interface to ensure policy consistency among distributed agents. Furthermore, the cloud layer monitors network dynamics in real time, achieving global policy optimization while protecting data privacy, ultimately improving the resource allocation efficiency and service quality robustness of multi-slice collaboration in complex heterogeneous environments.
[0073] In one specific embodiment of the present invention, each base station classifies services based on the service demand vectors reported by the terminals in real time. Specifically,
[0074] The business requirement vector is defined as follows: ;in, For the number of bits in the task, The number of CPU cycles required per bit. For the number of user connections;
[0075] Each base station performs service classification based on the aforementioned service demand vector, including:
[0076] when When the value exceeds a first preset threshold, the task is classified into an eMBB slice; where, Maximum number of bits for the task;
[0077] when When the value exceeds the second preset threshold, the task will be classified into a URLLC slice; where, The maximum number of CPU cycles required per bit;
[0078] when When the value exceeds the third preset threshold, the task will be classified into an mMTC slice; among which, This represents the maximum number of user connections.
[0079] It should be noted that when generating service requirements, the terminal device reports the service requirement vector to its base station in real time. This vector reporting is the data foundation for the minute-level coarse-grained slice management module to classify services. Its purpose is to provide the base station with a unified description of service characteristics, which facilitates rapid and accurate classification based on the core requirement differences of the three typical 5G service scenarios (eMBB, URLLC, and mMTC), thereby achieving targeted and efficient coarse-grained resource pre-allocation at the minute-level time scale.
[0080] Furthermore, by defining three key parameters of the business vector—task bit count, number of CPU cycles per bit, and number of user connections—to characterize the task's requirements for bandwidth, latency sensitivity, and connection scale, complete input information is provided for the classification step.
[0081] Furthermore, the user service classification module of each base station performs threshold judgment classification based on the service demand vector; by setting a first preset threshold, high bandwidth demand tasks (such as high-definition video transmission) are identified, ensuring that such tasks are allocated to the eMBB virtual slice framework focused on high-capacity transmission, providing high throughput guarantee for subsequent coarse-grained resource pre-allocation; by setting a second preset threshold, tasks highly sensitive to computational intensity and low latency (such as autonomous driving and industrial control) are identified, ensuring that such tasks enter the URLLC virtual slice framework emphasizing ultra-reliable low latency, thereby reserving dedicated resource paths for them at the minute level and avoiding competition with other services; by setting a third preset threshold, tasks with massive connection and low power consumption requirements (such as IoT sensors) are identified, ensuring that such tasks are classified into the mMTC virtual slice framework supporting high-density access, facilitating subsequent resource management using narrowband multiplexing and lightweight protocols.
[0082] In one specific embodiment of the present invention, coarse-grained resource pre-allocation is performed based on the classification results, and a virtual slicing framework is constructed. Specifically,
[0083] The coarse-grained resource pre-allocation obtains the basic resource quantities for various slices by solving a resource pre-allocation optimization problem that incorporates proportional fairness and exponential utility. The formula is:
[0084]
[0085]
[0086] in, , Given the spectrum allocated to slice s and the total amount of computing resources, For the historical average demand of slice s, The priority weights of slice s This is the balance coefficient between the proportional equity term and the index utility term. Let be the resource sensitivity coefficient of slice s. This refers to the total spectrum and computing resource capacity of the base station. The minimum guaranteed resource for slice s;
[0087] According to the solution We use hard slicing and / or soft slicing technologies to build independent virtual network slicing frameworks for the three types of slices and complete the initial resource binding.
[0088] It should be noted that the coarse-grained resource pre-allocation based on the classification results and the construction of a virtual slice framework aims to pre-allocate basic spectrum and computing resources for each type of slice within a minute-level cycle, according to the classified service requirements of eMBB, URLLC, and mMTC, and to construct an independent virtual network framework. This step ensures that each type of slice obtains a stable resource baseline through coarse-grained pre-allocation, avoiding resource starvation or fierce competition during second-level dynamic adjustments. This provides a reliable basic framework for subsequent second-level fine-grained dynamic resource adaptation, achieving stability and overall efficiency in resource allocation across two time scales.
[0089] Furthermore, by solving a resource pre-allocation optimization problem involving proportional fairness and exponential utility, the basic resource quantities for various slices are obtained. This aims to jointly maximize proportional fairness and resource satisfaction under the constraint of total resource capacity, ensuring that high-priority slices (such as URLLC) receive sufficient resource guarantees while also considering overall resource utilization efficiency. The solution results are directly used as input parameters for subsequent slice framework construction.
[0090] Furthermore, by employing hard slicing and / or soft slicing technologies to construct independent virtual network slicing frameworks for the three types of slices and completing initial resource binding, the abstract resource quantity R_s is transformed into a concrete virtual network structure: hard slicing provides strict guarantees through physical layer resource isolation, while soft slicing provides flexibility through logical reservation, thereby creating high-capacity transmission channels for eMBB, establishing low-latency dedicated paths for URLLC, and optimizing high-density access protocols for mMTC. This construction and binding process ensures initial isolation between slices while retaining a certain degree of resource boundary adjustability, providing an operable framework foundation for second-level dynamic resource adapters.
[0091] In one specific embodiment of the present invention, the Actor network and dual Critic network corresponding to all base stations are trained centrally in a global priority experience replay pool. Specifically,
[0092] The local observation information uploaded by each base station, the resource allocation actions performed, the rewards obtained, and the local observation information at the next moment are continuously collected as interactive experience data and stored in the global priority experience replay pool.
[0093] The local observation information of each base station n is defined as follows: ;in, This represents the local observation information of base station n at time t. These are the location, velocity, channel state, and service demand vector of user i, respectively. The set of users within the service range of base station n;
[0094] Construct the global state based on the local observation information of all base stations. and the coordinated actions of all base stations. .
[0095] It should be noted that the Actor networks and dual Critic networks corresponding to all base stations are trained centrally in the global priority experience replay pool. This aims to jointly optimize the global policy model of the MATD3 algorithm by utilizing the heterogeneous interaction experience data of all base stations through centralized training. This step realizes the core part of the centralized training-distributed execution framework, ensuring that each base station agent can still learn a high-quality policy that considers global cooperation even in a distributed decision-making environment that relies only on local observations. This suppresses non-stationarity issues in multi-agent scenarios and improves the convergence efficiency, stability, and robustness of overall resource allocation. This training process provides a high-quality Actor network parameter foundation for subsequent model parameter distribution and second-level fine-grained decision-making.
[0096] Furthermore, by collecting local observation information uploaded by each base station, the executed resource allocation actions, the obtained rewards, and the local observation information at the next time step as interactive experience data, this data is stored in a global priority experience replay pool. This aims to aggregate diverse experience samples across base stations, including high-value events such as sudden URLLC task preemption. A priority sampling mechanism is used to focus on learning rare but crucial interactive data, avoiding experience distribution bias. This collection and storage process provides a rich training dataset for subsequent parameter updates of the Critic and Actor networks, ensuring that the training process captures the potential correlations of complex dynamic environments.
[0097] Furthermore, by defining the local observation information of each base station, an independent and partially observable input state is provided for each base station agent, simulating the information limitations in actual distributed execution, while laying the foundation for constructing global information during centralized training in the cloud.
[0098] Furthermore, constructing a global state and joint actions of all base stations based on local observation information allows for the temporary creation of a global perspective during the cloud training phase. This perspective is used by the dual-Critic network to accurately evaluate the value of joint actions without requiring base stations to acquire global information during the execution phase. This achieves privacy protection and low-overhead multi-agent collaboration.
[0099] Furthermore, such as Figure 3 , 4 As shown, during the training phase, in order to generate the interactive experience data, each base station will output the deterministic actions from the Actor network. Combined with Gaussian noise, the exploration action is obtained. It operates within the current 5G network environment to generate instant rewards. The aforementioned reward This represents the computation time required for agent n to process all tasks, specifically expressed as... ,in This represents the proportion of computing resources allocated to user i. The purpose of this reward calculation is to quantify the impact of resource allocation actions on computational latency, guide the Critic network in evaluating the value of actions, and accelerate the convergence of the algorithm in heterogeneous dynamic environments by prioritizing high-value experiences (such as samples with significantly reduced computational latency) through a priority experience replay pool. This process is closely related to subsequent dual Critic and Actor updates, ensuring that the training experience data reflects real-world feedback.
[0100] Furthermore, after training is completed, the parameters issued include optimized Actor and dual Critic network parameters, ensuring that the base station has complete policy execution and evaluation capabilities.
[0101] In one specific embodiment of the present invention, the update of the dual-Critic network is specifically as follows:
[0102] Add truncated Gaussian noise to the target Actor output to generate exploratory target actions. The formula is:
[0103] ;
[0104] in, Represents the target Actor network. This represents the global state at the next moment. For the target Actor network parameters, It is Gaussian noise. For noise standards, The truncation range;
[0105] Based on the exploratory target action, the dual-Critic network calculates the target Q-value. The formula is:
[0106] ;
[0107] in, This represents the immediate reward received by base station n. As a discount factor, Describes the critic network for the i-th target. Let i be the network parameters for the i-th target Critic. For each base station, explore joint target actions;
[0108] The two Critic networks are synchronized by minimizing the temporal difference loss, as shown in the formula:
[0109] ;
[0110] in, Let r be the expected value of the Critic loss function, and r be the reward obtained. This refers to the online Q-value.
[0111] It should be noted that the update of the dual-Critic network aims to accurately evaluate the value function of the joint action through the dual-Critic structure and the target action noise mechanism, while suppressing the Q-value overestimation bias common in multi-agent environments. This update process ensures that the Critic network can provide reliable gradient guidance for the Actor network, thereby improving the convergence stability and robustness of the overall policy in complex dynamic heterogeneous environments. This step works closely with the subsequent delayed update of the Actor network and the soft update of the target network, together constituting the key improvement of the MATD3 algorithm over the traditional MADDPG.
[0112] Furthermore, adding truncated Gaussian noise to the target Actor output to generate exploratory target actions introduces controlled perturbations into the target actions, enhancing the policy's robustness to local observation errors and environmental non-stationarity, avoiding overfitting the current policy, and providing diverse evaluation samples for the dual Critic. This noise addition process directly generates the exploratory target actions required for subsequent target Q-value calculations.
[0113] Furthermore, based on the exploratory target action, the dual-critic network calculates the target Q-value y. This involves using two target criters to evaluate the value of the noisy target action separately, and taking the smaller value as a conservative estimate, effectively suppressing the overestimation bias of the Q-value. This calculation result is directly used as the update target for subsequent temporal difference loss.
[0114] Furthermore, the two Critic networks are updated synchronously by minimizing the temporal difference loss, which simultaneously optimizes the two online Critic networks to approach a conservative target Q-value, thereby improving the accuracy and stability of value assessment. This loss minimization process ensures that the two Critic networks maintain consistency during training and provides reliable guidance for the policy gradient of the Actor network.
[0115] In one specific embodiment of the present invention, the update of the Actor network specifically involves:
[0116] After the Critic network fully converges, the Actor network is updated once every preset number of steps, and the policy gradient is expressed as:
[0117] ;
[0118] in, Let the policy objective function be... Let be the expected gradient of the Actor's policy. As the first online Critic network, For the Actor network of base station n, Let n be the Actor network parameters for base station n;
[0119] The target Critic network is updated using a soft update mechanism, with the following formula:
[0120] ;
[0121] ;
[0122] in, This is the soft update coefficient.
[0123] It should be noted that the Actor network update involves optimizing the Actor network parameters to maximize long-term reward expectation, based on the reliable value assessment provided by the dual Critic network, while reducing the risk of policy oscillation through a delayed update mechanism. This update process ensures that the Actor network is only adjusted after the Critic network has fully converged, thereby improving the stability and efficiency of multi-agent cooperative strategies in the dynamic heterogeneous environment of 5G. This step, in conjunction with the preceding dual Critic update, forms the core improvement mechanism of the algorithm and provides an optimized Actor network for subsequent model parameter distribution.
[0124] Furthermore, after the Critic fully converges, the Actor network is updated once at a preset interval. The Q-value gradient evaluated by the Critic guides the Actor to adjust its strategy towards higher-value directions, avoiding oscillations caused by frequent updates, and relying only on local observations. To achieve distributed compatibility, this delayed update directly uses the result of the preceding Critic update as gradient input.
[0125] Furthermore, employing a soft update mechanism to update the target network progressively synchronizes the parameters of the online network and the target network, preventing drastic changes in the target value during training and improving overall convergence stability. This soft update process is executed after each Actor and Critic iteration, ensuring that the target network slowly tracks the improvements of the online network.
[0126] In one specific embodiment of the present invention, the latest Actor network parameters and dual Critic network parameters are combined with real-time collected local state observation information to perform deterministic policy reasoning, thereby obtaining and executing fine-grained resource adjustment actions. Specifically,
[0127] Using the issued Actor parameters Real-time local observation Deterministic policy reasoning yields the following fine-grained action representation:
[0128] ;
[0129] in, This represents the fine-grained resource adjustment action output by base station n at time t, including the spectrum allocation ratio for each user i. and computing resource allocation ratio ;
[0130] Will The mapping is to specific resource operations, including micro-slot pre-occupancy scheduling for URLLC slice users, carrier aggregation scheduling for eMBB slice users, and narrowband multiplexing scheduling for mMTC slice users.
[0131] When a momentary shortage of URLLC slice resources is detected, resources are preferentially borrowed from the free resource blocks of eMBB or mMTC slices, and the URLLC slice is increased in the next minute-level cycle. Compensation will be provided.
[0132] It should be noted that the deterministic policy inference using the latest Actor network parameters and real-time collected local state observation information involves leveraging the pre-trained Actor network parameters distributed from the cloud to generate fine-grained resource adjustment decisions based solely on locally collected real-time local observation information, and mapping these decisions to actual physical / virtual resource operations. This step implements the distributed execution portion of the MATD3 algorithm, ensuring real-time response to dynamic network changes (such as user movement, channel attenuation, and task bursts) on a minute-level coarse-grained framework, while also supporting cross-slice priority mechanisms, thereby improving overall resource utilization and differentiated service quality. This process bridges cloud training and edge execution, ensuring the system maintains stability and efficiency in highly dynamic environments.
[0133] Furthermore, each base station receives and distributes Actor network parameters. and dual Critic network parameters Subsequently, deterministic policy reasoning is mainly performed using the Actor parameter, while the dual Critic parameters can assist in local monitoring of action value or anomaly detection.
[0134] Furthermore, utilizing the issued Actor parameters Real-time local observation Deterministic policy reasoning involves generating abstract action vectors in real time based on local observations, avoiding the overhead of relying on global information, and fully utilizing global collaborative knowledge trained in the cloud to ensure that decisions take into account the coupling of multiple base stations. The reasoning result is directly used as input for subsequent mapping steps.
[0135] Furthermore, Mapping to specific resource operations involves transforming the abstract actions output by the Actor into executable physical resource adjustments. This ensures that URLLC obtains dedicated time slots with low interference, eMBB maximizes peak rates, and mMTC supports high-density, low-power access, thereby meeting the differentiated needs of each slice within a second-level timescale. This mapping process directly utilizes the action vectors generated by preceding inference as guidance.
[0136] Furthermore, the distribution of dual Critic parameters enables base stations to perform optional value assessments or further strategy fine-tuning locally, while maintaining consistency with cloud-based training and ensuring robust decision-making during the distributed execution phase. This distribution process is implemented through a secure interface, transmitting only necessary parameters to control communication overhead.
[0137] Furthermore, such as Figure 5 As shown, when a momentary shortage of URLLC slice resources is detected, resources are preferentially borrowed from the free resource blocks of eMBB or mMTC slices, and the corresponding slices are increased in the next minute-level cycle. The compensation mechanism aims to implement cross-slice priority preemption and dynamic compensation, ensuring that the ultra-low latency and high reliability of URLLC are not affected, while also preventing long-term resource loss in other slices. This preemption and compensation process is triggered in real time after the resource mapping operation, serving as a supplement to exception handling.
[0138] In a second embodiment of the present invention, the present invention provides a 5G network slicing resource allocation system based on a differentiated service time scale, such as... Figure 6 As shown, the system includes a slice management module 1, a centralized training module 2, and a decision execution module 3;
[0139] The slice management module 1 is used to classify the services of each base station according to the service demand vector reported by the terminal in real time within a minute-level time scale, and to pre-allocate coarse-grained resources and construct a virtual slice framework based on the classification results.
[0140] The centralized training module 2 is used to employ a multi-agent dual-delay deep deterministic policy gradient algorithm. Based on the interaction experience data continuously uploaded by each base station, it centrally trains the Actor network and dual Critic network corresponding to all base stations in the global priority experience replay pool. After the training converges or reaches the update cycle, it sends the latest Actor network parameters and dual Critic network parameters to each base station.
[0141] The decision execution module 3 is used to perform deterministic strategy reasoning on the latest Actor network parameters and dual Critic network parameters and real-time collected local state observation information within a second-level time scale, so as to obtain fine-grained resource adjustment actions and execute them.
[0142] In a third embodiment of the present invention, the present invention provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the 5G network slicing resource allocation method based on the differentiated service time scale as described above.
[0143] In Embodiment 4 of the present invention, the present invention provides a storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps in the 5G network slicing resource allocation method based on the differentiated service time scale as described above.
[0144] In summary, the 5G network slicing resource allocation method based on differentiated service time scales provided by this invention effectively solves the problems of unstable resource allocation and insufficient real-time performance in dynamic environments of existing technologies by introducing a hierarchical mechanism of minute-level coarse-grained resource pre-allocation and second-level fine-grained dynamic adjustment based on the MATD3 algorithm. At the same time, by adopting dual-critic minimum Q-value selection, target action noise perturbation, and delay policy updates, Q-value overestimation and policy oscillation are suppressed, improving the convergence efficiency and robustness of multi-agent cooperation. Thus, under the premise of ensuring differentiated QoS for various services, the overall optimization of network resource utilization and service quality is achieved.
[0145] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the modules described above can be referred to the corresponding process in the aforementioned method implementation, and will not be repeated here.
[0146] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0147] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.
[0148] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for allocating 5G network slicing resources based on differentiated service time scales, characterized in that: include, Within a minute-level timescale, each base station classifies services based on the service demand vectors reported by the terminals in real time, and performs coarse-grained resource pre-allocation and constructs a virtual slice framework based on the classification results; A multi-agent dual-delay deep deterministic policy gradient algorithm is adopted. Based on the interaction experience data continuously uploaded by each base station, the Actor network and dual Critic network corresponding to all base stations are trained in a global priority experience replay pool. After the training converges or the update cycle is reached, the latest Actor network parameters and dual Critic network parameters are sent to each base station. Within a second-level timescale, the latest Actor network parameters and dual Critic network parameters are combined with real-time collected local state observation information to perform deterministic policy reasoning, obtain fine-grained resource adjustment actions, and execute them.
2. The 5G network slicing resource allocation method based on differentiated service time scale according to claim 1, characterized in that: The step of classifying services by each base station based on the service demand vectors reported by the terminals in real time includes... The business requirement vector is defined as follows: ;in, For the number of bits in the task, The number of CPU cycles required per bit. For the number of user connections; Each base station performs service classification based on the aforementioned service demand vector, including: when When the value exceeds a first preset threshold, the task is classified into an eMBB slice; where, Maximum number of bits for the task; when When the value exceeds the second preset threshold, the task will be classified into a URLLC slice; where, The maximum number of CPU cycles required per bit; when When the value exceeds the third preset threshold, the task will be classified into an mMTC slice; among which, This represents the maximum number of user connections.
3. The 5G network slicing resource allocation method based on differentiated service time scale according to claim 2, characterized in that: The process of pre-allocating coarse-grained resources based on classification results and constructing a virtual slicing framework includes, The coarse-grained resource pre-allocation obtains the basic resource quantities for various slices by solving a resource pre-allocation optimization problem that incorporates proportional fairness and exponential utility. The formula is: ; ; in, , Given the spectrum allocated to slice s and the total amount of computing resources, For the historical average demand of slice s, The priority weights of slice s This is the balance coefficient between the proportional equity term and the index utility term. Let be the resource sensitivity coefficient of slice s. This refers to the total spectrum and computing resource capacity of the base station. The minimum guaranteed resource for slice s; According to the solution We use hard slicing and / or soft slicing technologies to build independent virtual network slicing frameworks for the three types of slices and complete the initial resource binding.
4. The 5G network slicing resource allocation method based on differentiated service time scale according to claim 3, characterized in that: The step of centrally training the Actor network and dual Critic network corresponding to all base stations in the global priority experience replay pool includes... The local observation information uploaded by each base station, the resource allocation actions performed, the rewards obtained, and the local observation information at the next moment are continuously collected as interactive experience data and stored in the global priority experience replay pool. The local observation information of each base station n is defined as follows: ;in, This represents the local observation information of base station n at time t. These are the location, velocity, channel state, and service demand vector of user i, respectively. The set of users within the service range of base station n; Construct the global state based on the local observation information of all base stations. and the coordinated actions of all base stations. .
5. The 5G network slicing resource allocation method based on differentiated service time scale according to claim 4, characterized in that: The update of the dual-Critic network includes, Add truncated Gaussian noise to the target Actor output to generate exploratory target actions. The formula is: ; in, Represents the target Actor network. This represents the global state at the next moment. For the target Actor network parameters, It is Gaussian noise. For noise standards, The truncation range; Based on the exploratory target action, the dual-Critic network calculates the target Q-value. The formula is: ; in, This represents the immediate reward received by base station n. As a discount factor, Describes the critic network for the i-th target. Let i be the network parameters for the i-th target Critic. For each base station, explore joint target actions; The two Critic networks are synchronized by minimizing the temporal difference loss, as shown in the formula: ; in, Let r be the expected value of the Critic loss function, and r be the reward obtained. This refers to the online Q-value.
6. The 5G network slicing resource allocation method based on differentiated service time scale according to claim 5, characterized in that: The updates to the Actor network include, After the Critic network fully converges, the Actor network is updated once every preset number of steps with a delay. The policy gradient is expressed as: ; in, Let the policy objective function be... Let be the expected gradient of the Actor's policy. As the first online Critic network, For the Actor network of base station n, Let n be the Actor network parameters for base station n; The target Critic network is updated using a soft update mechanism, with the following formula: ; ; in, This is the soft update coefficient.
7. The 5G network slicing resource allocation method based on differentiated service time scale according to claim 6, characterized in that: The process involves performing deterministic policy reasoning using the latest Actor network parameters and dual Critic network parameters, combined with real-time collected local state observation information, to derive and execute fine-grained resource adjustment actions. Using the issued Actor parameters Real-time local observation Deterministic policy reasoning yields the following fine-grained action representation: ; in, This represents the fine-grained resource adjustment action output by base station n at time t, including the spectrum allocation ratio for each user i. and computing resource allocation ratio ; Will The mapping is to specific resource operations, including micro-slot pre-occupancy scheduling for URLLC slice users, carrier aggregation scheduling for eMBB slice users, and narrowband multiplexing scheduling for mMTC slice users. When a momentary shortage of URLLC slice resources is detected, resources are preferentially borrowed from the free resource blocks of eMBB or mMTC slices, and the URLLC slice is increased in the next minute-level cycle. Compensation will be provided.
8. A 5G network slicing resource allocation system based on differentiated service time scales, characterized in that: It includes a slice management module, a centralized training module, and a decision execution module; The slice management module is used to classify the services of each base station according to the service demand vector reported by the terminal in real time within a minute-level time scale, and to pre-allocate coarse-grained resources and construct a virtual slice framework based on the classification results. The centralized training module is used to employ a multi-agent dual-delay deep deterministic policy gradient algorithm. Based on the interaction experience data continuously uploaded by each base station, it centrally trains the Actor network and dual Critic network corresponding to all base stations in the global priority experience replay pool. After the training converges or reaches the update cycle, the latest Actor network parameters and dual Critic network parameters are sent to each base station. The decision execution module is used to perform deterministic policy reasoning on the latest Actor network parameters and dual Critic network parameters and real-time collected local state observation information within a second-level time scale, so as to obtain fine-grained resource adjustment actions and execute them.
9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the 5G network slicing resource allocation method based on any one of claims 1-7.
10. A storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps in the 5G network slicing resource allocation method based on the differentiated service time scale as described in any one of claims 1-7.
