5G network slice dynamic resource allocation method based on deep reinforcement learning
By using a two-layer DDPG network with deep reinforcement learning in network slices to coordinate optimization of time-frequency resources and power, the problem of poor resource allocation in the prior art is solved, and more efficient resource utilization and better service quality are achieved.
Patent Information
- Application Number
- CN202510348734.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-24
- Publication Date
- 2025-06-20
AI Technical Summary
The prior art has low synchronous resource optimization rate in dynamic resource allocation of network slices and poor service quality compliance rate, resulting in poor allocation effect.
A two-layer DDPG network based on deep reinforcement learning is adopted to coordinate optimization of time-frequency resource allocation and power allocation, and the resource allocation network is trained through the dual-network collaborative training method to realize dynamic resource allocation.
It improves the allocation effect of dynamic resources of network slices and improves resource utilization and service quality compliance rate.
Smart Images
Figure CN120186622A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of network resource management, and more specifically, to a 5G network slice dynamic resource allocation method based on deep reinforcement learning. Background Art
[0002] With the acceleration of the 5G commercialization process and the deep penetration of the Internet of Things (IoT) infrastructure, IoT applications have evolved from single-device connections to complex scenario-based services. According to the prediction of the Global System for Mobile Communications Association, the number of global IoT connections will exceed 25 billion in 2025, giving rise to diversified vertical application scenarios such as vehicle-to-everything (V2X) and smart healthcare. These application scenarios have different requirements for network performance. For example, the peak rate of Enhanced Mobile Broadband (eMBB) needs to reach 10 Gbps, Ultra Reliable Low Latency Communications (URLLC) requires a latency of less than 1 ms, and Massive Machine Type Communication (mMTC) needs to support millions of device connections per square kilometer. Under such requirements, the traditional network architecture faces severe challenges, specifically manifested as follows: the resource efficiency reaches a bottleneck, that is, the static resource allocation strategy is difficult to cope with the spatio-temporal volatility of traffic, resulting in insufficient spectrum utilization to meet the demand; the Quality of Service (QoS) guarantee is in trouble, that is, the QoS requirements among heterogeneous services conflict with each other, such as the competition for latency and bandwidth resources between industrial control and video surveillance services; there are defects in security isolation, that is, there are problems of cross-service interference and attack surface spread under the background of sharing physical infrastructure.
[0003] As the core innovative technology of 5G technology, network slicing technology uses Network Functions Virtualization (NFV) and Software-defined Networking (SDN) to achieve virtualized segmentation of the physical network, providing a systematic solution to the above problems. Its technical advantages are reflected in: providing customized services, that is, independently configuring network functions, resource quotas, and QoS policies for each slice; flexibly orchestrating dynamic resources, that is, realizing elastic allocation of computing resources, storage resources, and spectrum resources based on real-time network status; achieving hard isolation guarantee, that is, realizing resource isolation between slices through the virtualization layer and SDN flow table rules, meeting the security isolation standard.
[0004] Although network slicing technology can solve the problems faced by traditional network architectures to a certain extent, there are still some problems in the actual application process. For example: the problem of dynamic resource optimization, that is, in a multi-slice competition environment, it is necessary to synchronously optimize multiple indicators such as resource utilization rate, operator revenue, and QoS compliance rate. However, the traditional convex optimization method commonly used to solve this problem has the problem of high computational complexity of the Pareto optimal solution; the problem of high-dimensional state space modeling: the network state dimension grows exponentially with the base station density and the number of slices. In this context, the computational complexity of the traditional Q-Learning algorithm will increase exponentially, and finally the problem of unable to perform resource allocation will occur.
[0005] In recent years, machine learning has become a technological hotspot for solving network slicing problems. Among them, algorithms of the reinforcement learning category have strong universality, and the algorithms optimized through a large amount of empirical training will be able to give better predictions. And in dealing with a large amount of diverse data, machine learning also has its unique advantages. The Deep Deterministic Policy Gradient (DDPG) algorithm, as an algorithm based on deep reinforcement learning (DRL), is particularly suitable for solving problems in continuous action spaces, and RAN resource allocation is a typical continuous control problem.
[0006] The prior art discloses a deterministic network slice resource allocation method based on information freshness, which combines network slicing technology and machine learning technology to allocate communication resources, including step 1: constructing a network slice resource allocation scenario in the industrial Internet of Things; step 2: based on the network slice resource allocation scenario, establishing corresponding transmission delay models, queuing delay models, and age of information models; step 3: according to the transmission delay models, queuing delay models, and age of information models, establishing an optimization problem for slice and resource allocation; step 4: solving the optimization problem in step 3 through the DDPG learning framework based on the actor-critic model to achieve dynamic allocation of network slice resources; step 5: solving the network slice resource allocation strategy with the largest Q value through the batch gradient descent method, so that under certain network resource constraints, the sum of the probabilities that the age of information of IoT devices exceeds a predetermined threshold is minimized, thus realizing real-time update of industrial Internet of Things information and reducing the delay and computational complexity of information transmission. However, this solution only optimizes a single indicator of delay, and the synchronous resource optimization rate is not high, and the quality of service is not good, resulting in poor allocation effects. Summary of the Invention
[0007] To solve the problems of low synchronization resource optimization rate, poor service quality compliance rate, and poor allocation effect in the current network slicing dynamic resource allocation technology, the present invention proposes a 5G network slicing dynamic resource allocation method based on deep reinforcement learning, which models multi-type service data and environmental data, and uses a double-layer DDPG network to achieve collaborative optimization of time-frequency resource allocation and power allocation, improving the allocation effect of network slicing dynamic resources.
[0008] To achieve the above technical effects, the technical solution of the present invention is as follows:
[0009] The present application proposes a 5G network slicing dynamic resource allocation method based on deep reinforcement learning, including the following steps:
[0010] S1. Perform basic modeling to generate input data required for resource allocation, including multi-type service data parameter modeling and environmental data parameter modeling; construct a resource allocation network, where the resource allocation network includes a time-frequency resource allocation DDPG network and a power allocation DDPG network;
[0011] S2. Based on the input data required for resource allocation generated by the basic modeling, obtain state space information, and use the method of double-network collaborative training to train the resource allocation network;
[0012] S3. After training is completed, when each service arrives, the time-frequency resource allocation DDPG network outputs a time-frequency resource allocation action according to the state space information. If the resource allocation is successful, the allocation result is used as a part of the state space information to update the state space information, which is input to the power allocation DDPG network. The power allocation DDPG network outputs a power allocation action for the service according to the updated state space information to obtain the final power allocation scheme.
[0013] In this technical solution, first, perform basic modeling including service data parameters and environmental data parameters and construct a resource allocation network using a double-layer DDPG network. Use the method of double-network collaborative training to train the resource allocation network. After training is completed, when each service arrives, the time-frequency resource allocation DDPG network outputs a time-frequency resource allocation action according to the state space information. If the resource allocation is successful, the allocation result is used as a part of the state space information to update the state space information, which is input to the power allocation DDPG network. The power allocation DDPG network outputs a power allocation action for the service according to the updated state space information to obtain the final power allocation scheme, realizing the collaborative optimization of time-frequency resource allocation and power allocation, and improving the allocation effect of network slicing dynamic resources.
[0014] Preferably, the multi-type service data parameter modeling in step S1 includes obtaining the service type and generating service information. The service types include: URLLC service, eMBB service, and mMTC service. The generated service information at least includes the size of the data packet to be transmitted and the service arrival time. For services with special QoS requirements, QoS requirements are generated.
[0015] The environmental data parameter modeling in step S1 includes channel modeling and other modeling. The channel modeling uses a block fading channel model, including small-scale fading and large-scale fading. Other modeling is related to the calculation and processing in the resource allocation network, including processing delay and queuing delay.
[0016] Preferably, the state space information in step S2 includes: service type, available resource status of the current time-frequency resource pool.
[0017] During training, first fix the power allocation DDPG network and train the time-frequency resource allocation DDPG network. Then fix the time-frequency resource allocation DDPG network and train the time-frequency resource allocation DDPG network, and alternately iterate the training until the resource allocation network converges.
[0018] For the power part involved in training the time-frequency resource allocation DDPG network, calculate according to the estimated transmission power based on the reference signal-to-noise ratio, the current channel state, and the currently allocable power. The estimated transmission power is less than the currently allocable power. At the beginning of training, the reference signal-to-noise ratio is a preset fixed value, and is updated subsequently by the power allocation output of the power allocation DDPG network.
[0019] Preferably, in step S3, when each service arrives, the time-frequency resource allocation DDPG network outputs a time-frequency resource allocation action according to the state space information. The process is as follows:
[0020] In each simulation time unit, when the time-frequency resource allocation DDPG network confirms that the current time is the allocation ready time for a certain service, it determines that the allocation requirement has arrived.
[0021] For each service with an arrived allocation requirement, output a time-frequency resource allocation action according to the state space information. After the time-frequency resource allocation action is output, detect the time-frequency resource allocation action according to the expected service QoS and the current resource pool state under the current time-frequency resource allocation action, and return the main reward value according to the detection result.
[0022] The time-frequency resource allocation process includes:
[0023] S31: Allocate time-frequency resources according to the priority order of different services and judge whether the time-frequency resources of different services are sufficient. If so, output the time-frequency resource allocation action to obtain the time-frequency resource allocation result; otherwise, execute S32.
[0024] S32: Determine whether the service is a URLLC service. If so, preempt the time-frequency resources of the eMBB service and execute S33; otherwise, queue the service and wait. After the queue arrives, re-enter it into the time-frequency resource allocation DDPG network.
[0025] S33: Determine whether the resources are sufficient after preempting the eMBB time-frequency resources. If so, the URLLC service performs resource preemption and outputs the resource allocation result; otherwise, queue the service and wait. After the queue arrives, re-enter it into the time-frequency resource allocation DDPG network.
[0026] Preferably, when the URLLC service preempts the resources of the eMBB service, for the eMBB service whose resources are preempted, at the beginning of the next allocation time slot, the base station sends downlink preemption indication information through the physical downlink control channel, instructing the eMBB service terminal of the preempted resources to discard the corresponding information, reorganize the resource allocation requirements of the eMBB service of the preempted resources, and input them into the resource allocation DDPG network.
[0027] Preferably, the process of detecting the time-frequency resource allocation action is as follows:
[0028] Judge whether the time-frequency resource allocation action is qualified. If so, return a positive action reward; otherwise, return a negative action reward. The unqualified time-frequency resource allocation actions include: the service QoS does not meet the standard, the eMBB service and the mMTC service illegally preempt the resources of other services, and the URLLC service still does not meet the QoS standard after preempting the resources. The unqualified time-frequency resource allocation actions are not actually executed, the resource allocation fails, the QoS of the service is corrected, and the reallocation time of the delayed service is determined according to the service type. The service with resource allocation failure enters the next resource allocation cycle. If a single service is delayed multiple times, a delay penalty term is introduced, and the absolute value of the penalty term increases non-linearly, and the more times of delay, the faster it grows;
[0029] When any action cannot meet the QoS of the service, a huge service transmission failure penalty term is introduced.
[0030] Preferably, the updated state space information includes service type, resource allocation result, user channel gain, and estimated transmission power;
[0031] When the resource allocation is successful, the power allocation DDPG network calculates the power allocation action for the current service according to the updated state space information. The power allocation action is the estimated change in the transmission power of the estimated transmission power for the current service in the time-frequency resource allocation DDPG network. The sum of the estimated transmission power and the estimated change in the transmission power is the final allocated power for the current service.
[0032] Preferably, after the power allocation action is output, it is determined whether the power allocation action is qualified. If it is qualified, the modulation method and coding efficiency are updated, and the service QoS is updated. Otherwise, the power allocation action mapping is performed;
[0033] The qualified power allocation action is: the final allocated power makes the service QoS reach the standard and is less than the available power; the unqualified power allocation actions include: the final allocated power is small, resulting in the service QoS not reaching the standard; the final allocated power is greater than the available power.
[0034] Preferably, the power allocation action mapping includes: when the allocated power is small and the service QoS does not reach the standard, the power is mapped to the minimum power that can meet the service QoS requirement, and the minimum power is obtained by inverse solution using the channel capacity model, bandwidth, and noise level; when the allocated power is greater than the allocable power and is unqualified, the power is mapped to the maximum power that can be allocated;
[0035] If the power allocation action is mapped, an over - quantity penalty term is introduced to punish the power allocation action. Otherwise, a reward is obtained. The reward is based on the revenue curve and cost curve of each service. After subtraction and multiplication by a preset scaling factor, the revenue of a single service is obtained. The weighted sum of the revenues of all current services is the reward value of the current action. Among them, both the revenue curve and the cost curve are based on basic modeling. The revenue curve is mainly determined by QoS, and the cost curve is mainly determined by the transmission power.
[0036] Preferably, after the final power allocation scheme is determined, the signal - to - noise ratio is calculated through the finally allocated power and the channel state. If the power allocation action mapping does not occur, the final signal - to - noise ratio, channel state, power before allocation, service type, original size of the data packet to be transmitted, and final revenue are all stored in the reference signal - to - noise ratio experience pool for the selection of the reference signal - to - noise ratio in the time - frequency resource allocation DDPG network.
[0037] The present invention proposes a 5G network slice dynamic resource allocation method based on deep reinforcement learning. First, basic modeling including service data parameters and environmental data parameters is performed, and a resource allocation network is constructed using a double - layer DDPG network. The resource allocation network is trained using a method of co - training of double networks. After training is completed, when each service arrives, the time - frequency resource allocation DDPG network outputs a time - frequency resource allocation action according to the state - space information. If the resource allocation is successful, the allocation result is used as a part of the state - space information to update the state - space information and input it into the power allocation DDPG network. The power allocation DDPG network outputs the power allocation action of the service according to the updated state - space information, and a final power allocation scheme is obtained, realizing the collaborative optimization of time - frequency resource allocation and power allocation, and improving the allocation effect of network slice dynamic resources. Description of the Drawings
[0038] Figure 1 It shows a schematic flowchart of a 5G network slice dynamic resource allocation method based on deep reinforcement learning proposed in Embodiment 1 of the present invention;
[0039] Figure 2 It shows a schematic content diagram of the basic modeling proposed in Embodiment 2 of the present invention;
[0040] Figure 3 It shows a schematic flowchart of time-frequency resource allocation in Embodiment 2 of the present invention;
[0041] Figure 4 It shows a schematic flowchart of power resource allocation proposed in Embodiment 3 of the present invention. Detailed implementation manners
[0042] The accompanying drawings are only for illustrative purposes and should not be construed as a limitation of this patent;
[0043] To better illustrate this embodiment, some parts of the accompanying drawings will be omitted, enlarged or reduced, and do not represent the actual size;
[0044] For those skilled in the art, it is understandable that some well-known content descriptions in the accompanying drawings may be omitted.
[0045] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0046] The description of the positional relationship in the accompanying drawings is only for illustrative purposes and should not be construed as a limitation of this patent;
[0047] Embodiment 1
[0048] This embodiment proposes a 5G network slice dynamic resource allocation method based on deep reinforcement learning. The schematic flowchart of this method is shown in Figure 1 , and includes the following steps:
[0049] S1. Perform basic modeling to generate the input data required for resource allocation, including multi-type service data parameter modeling and environmental data parameter modeling; construct a resource allocation network, and the resource allocation network includes a time-frequency resource allocation DDPG network and a power allocation DDPG network;
[0050] S2. Based on the input data required for resource allocation generated by the basic modeling, obtain the state space information, and train the resource allocation network by using the method of dual-network collaborative training;
[0051] S3. Training is completed. When each service arrives, the time-frequency resource allocation DDPG network outputs a time-frequency resource allocation action according to the state space information. If the resource allocation is successful, the allocation result is used as part of the state space information to update the state space information, which is then input into the power allocation DDPG network. The power allocation DDPG network outputs the power allocation action for the service according to the updated state space information, and finally obtains the power allocation scheme.
[0052] In this embodiment, first, basic modeling including service data parameters and environmental data parameters is performed, and a resource allocation network is constructed using a double-layer DDPG network. The resource allocation network is trained using the method of co-training of dual networks. After the training is completed, when each service arrives, the time-frequency resource allocation DDPG network outputs a time-frequency resource allocation action according to the state space information. If the resource allocation is successful, the allocation result is used as part of the state space information to update the state space information, which is then input into the power allocation DDPG network. The power allocation DDPG network outputs the power allocation action for the service according to the updated state space information, and finally obtains the power allocation scheme, realizing the collaborative optimization of time-frequency resource allocation and power allocation, and improving the allocation effect of network slice dynamic resources.
[0053] Embodiment 2
[0054] In this embodiment, the modeling of multi-type service data parameters described in step S1 includes obtaining the service type and generating service information. The service types include: URLLC service, eMBB service, and mMTC service. The generated service information includes at least the size of the data packet to be transmitted and the service arrival time. For services with special QoS requirements, QoS requirements are generated.
[0055] Specifically, the QoS requirement for the service Mobile Operation Panel:Motion control is that the maximum end-to-end delay needs to be less than 1 ms; the QoS requirement for the service Mobile Operation Panel:Safety data stream is that the maximum end-to-end delay needs to be less than 10 ms; the QoS requirement for the service Mobile robots is determined by the transmission interval of this service instance.
[0056] The modeling of environmental data parameters described in step S1 includes channel modeling and other modeling. The channel modeling uses the block fading channel model, which includes two parts: small-scale fading and large-scale fading. Other modeling is related to the calculation and processing in the resource allocation network, including processing delay and queuing delay.
[0057] For the content schematic diagram of the basic modeling including the modeling of multi-type service data parameters and environmental data parameters, see Figure 2 .
[0058] Specifically, the URLLC service is an ultra - reliable and low - latency communication service. The service information also includes: the end - to - end latency requirement is less than 1 ms, the user - plane latency requirement is less than 0.5 ms, the reliability requirement is that the bit error rate is less than 10 -5 , the scheduling method is OFDM symbol - level scheduling, the minimum granularity is 2 OFDM symbols, the sub - carrier spacing is a dynamic sub - carrier spacing with a range of 30 kHz - 120 kHz, the calculation method of rate tolerance is based on the Shannon formula for finite symbols, data transmission rate range, required bandwidth, minimum number of device accesses per square kilometer, and other parameters;
[0059] Specifically, for the URLLC service in the action control plane, the service information also includes a packet size of 10 - 100 bytes, a transmission interval of less than 1 ms, an end - to - end latency of less than 1 ms, the number of users being greater than or equal to 2, a service rate of 12 Mbit / s - 16 Mbit / s, the packet distribution being a truncated log - normal distribution, and the packet arrival process being a Poisson process;
[0060] Specifically, the eMBB service is an enhanced mobile broadband service. The service information also includes: a service transmission rate of 50 Mbps, a slot - level scheduling method, a fixed sub - carrier spacing of 30 kHz, the calculation method of rate tolerance being the Shannon formula, data transmission rate range, required bandwidth, minimum number of device accesses per square kilometer, and other parameters;
[0061] Specifically, for the eMBB service in the video stream, the service information includes: a packet size of 20 - 250 bytes, and the packet size and arrival conforming to a truncated Pareto distribution;
[0062] Specifically, the mMTC service is a massive machine - type communication service. The service information also includes: device density, which is the number of devices per square kilometer, a packet size of 32 bytes, and the service arrival conforming to a beta distribution with shape parameters taken as 3 and 4 respectively;
[0063] Specifically, channel modeling uses a block - fading channel model. Resource allocation is carried out in units of physical resource blocks, and the channel gain is calculated. The formula is:
[0064] g u (t)= u |h u (t)| 2
[0065]
[0066] where h u (t) is the complex coefficient of small - scale fading, h u(t) to CN(0,1), a u is the large-scale fading path loss, L(d0) is the free space path loss in dB, the reference transmission is related to the distance 0, d u is the actual distance from user u to the base station, is a Gaussian random variable with a mean of 0 and a standard deviation of σ u ;
[0067] Specifically, the processing delay is calculated using stochastic network calculus. An arrival curve A(t) and a service curve S(t) are established. The moment-generating function (MGF) is used to introduce a degree-of-freedom parameter δ to determine the stochastic process. The Exponentially Bounded Burstiness (EBB) model is adopted to introduce an overflow parameter and a sample path parameter. The EBB and MGF models are connected through the Chernoff bound to calculate the overflow probability;
[0068] Specifically, the queuing delay is determined by the time difference between the arrival time of the service demand and the actual resource allocation time.
[0069] Specifically, other modeling also includes the calculation of the air interface delay. The process is to calculate the required number of OFDM symbols according to the packet size, modulation order, and coding efficiency; calculate the air interface delay in combination with the service type, bandwidth, and subcarrier spacing;
[0070] Specifically, other modeling also includes setting the processing model of the service. The available processing models include: first-come, first-served, random processing, priority sorting (URLLC services have priority over other services, and the priority within the same type of service is determined by the priority preset by the service), hybrid methods (URLLC services are preferentially allocated, eMBB services are randomly allocated, etc.).
[0071] In this embodiment, the state space information described in step S2 includes: service type, available resource status of the current time-frequency resource pool, channel status
[0072] During training, first fix the power allocation DDPG network and train the time-frequency resource allocation DDPG network. Then fix the time-frequency resource allocation DDPG network and train the time-frequency resource allocation DDPG network. Alternately iterate the training until the resource allocation network converges;
[0073] Specifically, the method of dual-network collaborative training also includes an experience replay mechanism. The experience replay mechanism uses priority experience replay to store state transition samples, and the priority is dynamically adjusted by the temporal difference error;
[0074] Specifically, the method of dual-network collaborative training also includes adopting a soft update strategy to slowly synchronize the main network parameters to the target network;
[0075] For the power part involved in training the DDPG network for time-frequency resource allocation, it is calculated based on the estimated transmission power according to the reference signal-to-noise ratio, the current channel state, and the currently allocable power. The estimated transmission power is less than the currently allocable power. At the start of training, the reference signal-to-noise ratio is a preset fixed value and is updated subsequently by the power allocation output of the power allocation DDPG network.
[0076] Specifically, the power part specifically refers to the signal-to-noise ratio part. Power allocation is directly related to the magnitude of the signal-to-noise ratio. The signal-to-noise ratio determines the type of coding method and the coding efficiency, thereby affecting the number of OFDM symbols for transmitting information.
[0077] In this embodiment, in step S3, when each service arrives, the time-frequency resource allocation DDPG network outputs a time-frequency resource allocation action according to the state space information. The process is as follows:
[0078] In each simulation time unit, when the time-frequency resource allocation DDPG network confirms that the current time is the allocation ready time for a certain service, it determines that the allocation requirement has arrived.
[0079] For each service with an arrived allocation requirement, according to the state space information, a time-frequency resource allocation action is output. After the time-frequency resource allocation action is output, according to the expected service QoS under the current time-frequency resource allocation action and the current resource pool state, the time-frequency resource allocation action is detected, and according to the detection result, the main reward value is returned.
[0080] Specifically, the state space information includes: service type, the available resource state of the current time-frequency resource pool, and the channel state. Among them, the service type includes information such as whether there is a service requirement arrival, the type of service to be transmitted, the data size, and the maximum delay requirement, which is determined by the service model / service data used in the basic modeling part and can be various service models under the three network slice services, such as the motion control plane service in the URLLC slice, the Mobile robots service, etc.; the available resource state of the current time-frequency resource pool includes information such as the positions of free and occupied resource blocks and the occupied service type; the channel state includes the current channel quality information.
[0081] The flow schematic diagram of the time-frequency resource allocation is shown in Figure 3 , including:
[0082] S31: Allocate time-frequency resources according to the priority order of different services and determine whether the time-frequency resources of different services are sufficient. If so, output the time-frequency resource allocation action to obtain the time-frequency resource allocation result; otherwise, execute S32. In this embodiment, for different services URLLC, eMBB, and mMTC, the priority order is URLLC > eMBB > mMTC.
[0083] S32: Determine whether the service is a URLLC service. If so, preempt the time-frequency resources of the eMBB service and execute S33; otherwise, perform service queuing and waiting. After the queuing is completed, re-enter it into the time-frequency resource allocation DDPG network. If all time-frequency resources are insufficient, based on the principle that the high-priority service URLLC triggers the preemption of the time-frequency resources of the low-priority service eMBB service, perform resource preemption.
[0084] S33: Determine whether the time-frequency resources are sufficient after preempting the eMBB time-frequency resources. If so, the URLLC service performs time-frequency resource preemption and outputs the time-frequency resource allocation result; otherwise, perform service queuing and waiting. After the queuing is completed, re-enter it into the time-frequency resource allocation DDPG network.
[0085] In this embodiment, when the URLLC service preempts the resources of the eMBB service, for the eMBB service whose resources are preempted, at the beginning of the next allocation time slot, the base station sends a downlink preemption indication message through the physical downlink control channel, instructing the eMBB service terminal of the preempted resources to discard the corresponding information, reorganize the resource allocation requirements of the eMBB service of the preempted resources, and input them into the resource allocation DDPG network.
[0086] Specifically, 14 OFDM symbols can be sent in the time domain for each allocation time slot. The base station is a 5G base station, and the physical downlink control channel is a physical layer channel used to transmit key control information in the communication system; the shrunk-downlink preemption indication message is a key control signal used to dynamically adjust resource allocation, indicating that some of the already allocated resources are preempted by a higher-priority service and need to be immediately released or avoided for use.
[0087] In this embodiment, the process of detecting the time-frequency resource allocation action is as follows:
[0088] Determine whether the time-frequency resource allocation action is qualified. If so, return a positive action reward; otherwise, return a negative action reward. The unqualified time-frequency resource allocation actions include: the service QoS not meeting the standard, the eMBB service and the mMTC service illegally preempting the resources of other services, and the URLLC service still having unqualified QoS after preempting the resources. The unqualified time-frequency resource allocation actions are not actually executed, the resource allocation fails, the QoS of the service is corrected, and the reallocation time of the delayed service is determined according to the service type. The service with resource allocation failure enters the next resource allocation cycle. If a single service is delayed multiple times, a delay penalty term is introduced, and the absolute value of the penalty term increases non-linearly, with the growth rate increasing faster as the number of delays increases.
[0089] Specifically, for the URLLC service, the resource allocation cycle is every two OFDM symbols, and for the eMBB service, the resource allocation cycle is every 14 OFDM symbols.
[0090] When any action fails to meet the QoS of the service, a huge service transmission failure penalty term is introduced.
[0091] Specifically, the size of the action reward is controlled by the main reward value A, the resource preemption reward correction term B, and the service delay reward correction term C. For the main reward value A, a reward is obtained each time an action is executed. If the action meets the service quality of the current service, the main reward value A is positive; otherwise, it is negative. According to the service priority of URLLC > eMBB > mMTC, the absolute value of the main reward value A decreases gradually. Among the same type of services, the absolute value of the reward value of the high-priority service is larger.
[0092] For the resource preemption reward correction term B, it is positive, and the initial value is less than the difference between the absolute values of the URLLC and eMBB reward values. If a single eMBB service is preempted multiple times, B decreases exponentially.
[0093] For the service delay reward correction term C, it is negative, and for the same service, the more times it is delayed, the faster it decreases.
[0094] Specifically, the huge service transmission failure penalty term is a relative concept, a negative penalty term one order of magnitude larger than the relative negative main reward value.
[0095] In this embodiment, the updated state space information includes service type, time-frequency resource allocation result, user channel gain, and estimated transmission power.
[0096] When the time-frequency resource allocation is successful, the power allocation DDPG network calculates the power allocation action for the current service according to the updated state space information. For the schematic diagram of the power allocation process, see Figure 4The power allocation action is the estimated change in the transmission power estimated in the time-frequency resource allocation DDPG network for the current service. The sum of the estimated transmission power and the estimated change in transmission power is the final allocated power for the current service.
[0097] In this embodiment, after the power allocation action is output, it is determined whether the power allocation action is qualified. If it is qualified, the modulation method and coding efficiency are updated, and the service QoS is updated. Otherwise, a power allocation action mapping is performed.
[0098] The qualified power allocation action is: the final allocated power enables the service QoS to meet the standard and is less than the available power. The unqualified power allocation actions include: the final allocated power is small, resulting in the service QoS not meeting the standard; the final allocated power is greater than the available power.
[0099] In this embodiment, the power allocation action mapping includes: when the allocated power is small and the service QoS does not meet the standard, the power is mapped to the minimum power that can meet the service QoS requirements, and the minimum power is obtained by inverse solution using the channel capacity model, bandwidth, and noise level; when the allocated power is greater than the allocable power and is unqualified, the power is mapped to the maximum power that can be allocated.
[0100] If there is a power allocation action mapping, an overage penalty term is introduced to penalize the power allocation action. Otherwise, a reward is obtained. The reward is based on the revenue curve and cost curve of each service. After subtraction and multiplication by a preset scaling factor, the revenue of a single service is obtained. The weighted sum of the revenues of all current services is the reward value of the current action. Among them, both the revenue curve and the cost curve are based on basic modeling. The revenue curve is mainly determined by QoS, and the cost curve is mainly determined by the transmission power.
[0101] In this embodiment, after the final power allocation scheme is determined, the signal-to-noise ratio is calculated through the finally allocated power and the channel state. If there is no power allocation action mapping, the final signal-to-noise ratio, channel state, allocated power before power allocation, service type, original size of the data packet to be transmitted, and final revenue are all stored in the reference signal-to-noise ratio experience pool for reference signal-to-noise ratio selection in the time-frequency resource allocation DDPG network.
[0102] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all the implementation manners here. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A 5G network slice dynamic resource allocation method based on deep reinforcement learning, characterized in that: The following steps are involved: S1. Perform basic modeling to generate input data required for resource allocation, including multi-type business data parameter modeling and environmental data parameter modeling; construct a resource allocation network, the resource allocation network including a time-frequency resource allocation DDPG network and a power allocation DDPG network; S2. Based on the input data required for resource allocation generated by basic modeling, the state space information is obtained, and the resource allocation network is trained using a dual network collaborative training method; S3. Training is completed. When each service arrives, the time-frequency resource allocation DDPG network outputs a time-frequency resource allocation action based on the state space information. If the resource allocation is successful, the allocation result is used as part of the state space information to update the state space information and input it to the power allocation DDPG network. The power allocation DDPG network outputs the power allocation action of the service based on the updated state space information to obtain the final power allocation plan.
2. According to a 5G network slice dynamic resource allocation method based on deep reinforcement learning according to claim 1, it is characterized in that: The multi-type service data parameter modeling described in step S1 includes obtaining service types and generating service information, wherein the service types include: URLLC service, eMBB service and mMTC service, and the generated service information includes at least the size of the data packet to be transmitted and the service arrival time. For services with special QoS requirements, QoS requirements are generated; The environmental data parameter modeling described in step S1 includes channel modeling and other modeling. The channel modeling adopts a block fading channel model, including small-scale fading and large-scale fading. Other modeling is related to the calculation processing in the resource allocation network, including processing delay and queuing delay.
3. According to a 5G network slice dynamic resource allocation method based on deep reinforcement learning according to claim 2, it is characterized in that: The state space information described in step S2 includes: service type, available resource status of the current time-frequency resource pool; During training, first fix the power allocation DDPG network and train the time-frequency resource allocation DDPG network. Then fix the time-frequency resource allocation DDPG network and train the time-frequency resource allocation DDPG network. Alternate and iterate the training until the resource allocation network converges. The power part involved in training the time-frequency resource allocation DDPG network is calculated based on the reference signal-to-noise ratio, the current channel status and the estimated transmit power of the current allocatable power. The estimated transmit power is less than the current allocatable power. At the beginning of training, the reference signal-to-noise ratio is a preset fixed value, which is subsequently updated by the power allocation output of the power allocation DDPG network.
4. According to a 5G network slice dynamic resource allocation method based on deep reinforcement learning according to claim 3, it is characterized in that: In step S3, when each service arrives, the DDPG network for time-frequency resource allocation outputs a time-frequency resource allocation action according to the state space information. The process is as follows: In each simulation time unit, when the time-frequency resource allocation DDPG network confirms that the current time is the allocation ready time of a certain service, it is determined that the allocation demand has arrived; For each service that has an allocation demand, the time-frequency resource allocation action is output according to the state space information. After the time-frequency resource allocation action is output, the time-frequency resource allocation action is detected according to the expected service QoS and the current resource pool status under the current time-frequency resource allocation action, and the main reward value is returned according to the detection result; The time-frequency resource allocation process includes: S31: judging whether the time-frequency resources of different services are sufficient according to the priority order of different services, and if so, outputting the time-frequency resource allocation action to obtain the time-frequency resource allocation result; Otherwise, execute S32; S32: Determine whether the service is a URLLC service. If so, seize the time and frequency resources of the eMBB service and execute S33; otherwise, queue the service and wait for it. After the queue arrives, re-enter the service into the DDPG network for time and frequency resource allocation; S33: After seizing the eMBB time-frequency resources, determine whether the resources are sufficient. If so, the URLLC service will seize the resources and output the resource allocation result; otherwise, the service will queue up and wait, and after the queue arrives, it will be re-entered into the time-frequency resource allocation DDPG network.
5. According to a 5G network slice dynamic resource allocation method based on deep reinforcement learning according to claim 4, it is characterized in that: When the URLLC service preempts eMBB service resources, for the eMBB service whose resources are preempted, at the beginning of the next allocated time slot, the base station sends downlink preemption indication information through the physical downlink control channel, instructing the eMBB service terminal whose resources are preempted to discard the corresponding information, reorganize the resource allocation requirements of the eMBB service whose resources are preempted, and input them into the resource allocation DDPG network.
6. According to a 5G network slice dynamic resource allocation method based on deep reinforcement learning according to claim 5, it is characterized in that: The process of detecting the time-frequency resource allocation action is as follows: Determine whether the time-frequency resource allocation action is qualified. If so, return a positive action reward; otherwise, return a negative action reward; the unqualified time-frequency resource allocation actions include: service QoS not meeting the standard, eMBB service and mMTC service illegally occupying resources of other services, and URLLC service still not meeting the QoS standard after occupying resources; unqualified time-frequency resource allocation actions are not actually executed, resource allocation fails, QoS correction is performed on the service, and the reallocation time of the delayed service is determined according to the service type. The service that fails to allocate resources enters the next resource allocation cycle. If a single service is delayed multiple times, a delay penalty item is introduced, and the absolute value of the penalty item increases nonlinearly, and the more delays, the faster the increase; When any action cannot meet the QoS of the service, a huge service transmission failure penalty item is introduced.
7. According to claim 6, a 5G network slice dynamic resource allocation method based on deep reinforcement learning is characterized in that: The updated state space information includes service type, time-frequency resource allocation result, user channel gain, and estimated transmission power; When the time-frequency resource allocation is successful, the power allocation DDPG network calculates the power allocation action for the current service based on the updated state space information. The power allocation action is the estimated transmit power change of the estimated transmit power for the current service in the time-frequency resource allocation DDPG network. The estimated transmit power plus the estimated transmit power change is the final allocated power for the current service.
8. According to claim 7, a 5G network slice dynamic resource allocation method based on deep reinforcement learning is characterized in that: After the power allocation action is output, determine whether the power allocation action is qualified, if qualified, update the modulation mode and coding efficiency, and update the service QoS, otherwise, perform power allocation action mapping; The qualified power allocation action is: the final allocated power makes the service QoS meet the standard and is less than the available power; The unqualified power allocation actions include: the last allocated power is small, resulting in the service QoS not meeting the standard; the last allocated power is greater than the available power.
9. A 5G network slice dynamic resource allocation method based on deep reinforcement learning according to claim 8, characterized in that: The power allocation action mapping includes: when the allocated power is small and the service QoS does not meet the standard, the power is mapped to the minimum power that can meet the service QoS requirements, and the minimum power is obtained by inverse solution using the channel capacity model, bandwidth and noise level; when the allocated power is greater than the allocable power and is unqualified, the power is mapped to the maximum power that can be allocated; If the power allocation action is mapped, an excess penalty item is introduced to punish the power allocation action. Otherwise, a reward is obtained. The reward is based on the revenue curve and cost curve of each service. After subtraction, it is multiplied by the preset scaling factor to obtain the revenue of a single service. The weighted sum of all current service revenues is the reward value of the current action. Among them, the revenue curve and the cost curve are both basic modeling. The revenue curve is mainly determined by QoS, and the cost curve is mainly determined by the transmission power.
10. A 5G network slice dynamic resource allocation method based on deep reinforcement learning according to claim 9, characterized in that: After the final power allocation scheme is determined, the signal-to-noise ratio is calculated using the final allocated power and channel status. If no power allocation action mapping occurs, the final signal-to-noise ratio, channel status, allocated power before power allocation, service type and original data packet size to be transmitted, and final revenue are all stored in the reference signal-to-noise ratio experience pool for reference signal-to-noise ratio selection in the time-frequency resource allocation DDPG network.