DTN-based IoT RAN slice switching control and resource allocation method and system
By adopting the reinforcement learning algorithm of the digital twin network and actor-criticist framework in the IoT RAN slicing, real-time monitoring and resource allocation are realized, and data quality and timeliness in the IoT RAN slicing switching control and resource allocation are solved. Improves the optimization effect of system satisfaction and switching costs.
Patent Information
- Application Number
- CN202310171314.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-27
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-02-27
AI Technical Summary
The prior art has failed to effectively solve the problems of poor data quality, low value density and poor timeliness in IoT RAN slice switching control and resource allocation, and has failed to consider both system satisfaction and user switching costs.
Using a digital twin network (DTN)-based architecture, combined with a reinforcement learning algorithm of the actor-criticist framework, real-time monitoring and resource allocation are achieved through neural networks and in-point method optimizers to ensure the optimization of user satisfaction and switching costs.
Real-time switching control and resource allocation of IoT RAN slices is realized, system satisfaction and long-term effectiveness are improved, and data quality and timeliness in traditional methods are overcome.
Smart Images

Figure CN116155736B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to mobile communication technology, and in particular to a DTN-based Internet of Things RAN slice switching control and resource allocation method and system. Background Art
[0002] With the rapid development of IoT technology, a series of new vertical services and applications have emerged, such as smart transportation, high-precision manufacturing, virtual reality, and smart cities. However, the capacity of existing networks is far from enough to accommodate new vertical applications. Therefore, artificial intelligence and network slicing-assisted communication technologies have been developed. As a three-layer association of Internet of Things Devices (IoTE)-Network Slices (NS)-Base Stations (BS) in Radio Access Network (RAN) slices, switching control and resource allocation have become an important but complex issue.
[0003] Although traditional network management data and other extraction methods can obtain a large amount of network data, they have disadvantages such as poor data quality, low value density and poor timeliness, which hinder the real-time integration of the current network and RAN slice status. The digital twin network (DTN) can monitor user traffic, user requests, channel status, and network architecture at different time intervals. On the other hand, traditional static user access and resource allocation cannot adapt to the dynamically changing wireless access side. In the prior art, some people have proposed a dynamic switching control method for RAN slices based on deep Q network (DQN) to ensure the maximum rate and the minimum switching cost, but resource allocation is not considered, resulting in low system satisfaction.
[0004] From the above analysis, it can be clearly seen that the existing technology does not have a switching control and resource allocation solution for IoT RAN slices based on DTN that takes into account both system satisfaction and user switching costs. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a DTN-based IoT RAN slice switching control and resource allocation method, so that it can monitor the operation status of IoT devices in real time through DTN, overcome the shortcomings of poor quality, low value density and poor timeliness of existing network management data, and at the same time realize switching control and resource allocation through an improved actor-critic framework-based method.
[0006] To achieve the above purpose, the specific technical solutions adopted by the present invention are as follows:
[0007] A DTN-based IoT RAN slice switching control and resource allocation method, the key of which is to include the following steps:
[0008] S1. Construct a DTN-based IoT RAN slice architecture, wherein the DTN-based IoT RAN slice architecture includes a physical entity layer and a digital twin layer, and utilize the digital twin layer to monitor and collect the operating status of user devices, network slices, and base stations in the physical entity layer in real time, and trigger a switching instruction when the user satisfaction is lower than a preset threshold or violates the user service quality;
[0009] S2: After receiving the switching instruction, the central controller determines the switching action through the neural network in the actor home component according to the current network status;
[0010] S3: using the switching action determined in step S2 as an input of an interior point optimizer in the actor component, and the interior point optimizer outputs a resource allocation action, and the central controller executes the switching action and the resource allocation action as a joint action;
[0011] S4: input the switching action determined in step S2, the resource allocation action determined in step S3, and the rewards and network parameters formed by the system according to the execution of the joint action into the experience replay pool, use the critic component to evaluate the experience data in the experience replay pool, and input the evaluation results into the actor component for neural network training;
[0012] S5: Repeat steps S2-S4 until the functional model in the digital twin layer maintains convergence.
[0013] Optionally, the physical entity layer includes a variety of IoT user devices and base stations, and the digital twin layer includes a data storage device, a basic model, and a functional model, wherein:
[0014] The basic model is used to monitor the real network in the physical entity layer and build a digital twin of the real network using the collected network data;
[0015] The data storage is used to store the network data collected by the basic model;
[0016] The functional model is used to determine the switching action and the resource allocation action according to the network operation status in the current basic model.
[0017] Optionally, in step S2, after receiving the switching instruction, the central controller performs the following operations according to the currently observed network status:
[0018]
[0019] Switching actions output by the neural network in the actor component in: Indicates whether user u accesses slice n through base station b at time t-1. If yes, it is 1, otherwise it is 0; W b,n(t) represents the spectrum resources of slice n in base station b at time t; They represent the minimum rate and maximum delay requested by user u at time t respectively; They represent the minimum rate and maximum delay that slice n can provide at time t, respectively. t1 It represents the joint action of selecting base station b and slice n at time t, B is the set of base stations in the physical layer, N is the set of network slices in the physical layer, and U is the set of user equipment in the physical layer.
[0020] Optionally, in step S3, the switching action of the neural network output in the actor component Input the interior point optimizer, and the interior point optimizer outputs the resource allocation action represents the spectrum resources allocated to user u by slice n in base station b at time t, and the central controller performs joint action a t =(a t1 ,a t2 ) and enter the next state s t+1 , and get reward r t , and the empirical data (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool.
[0021] Optionally, in step S4, the neural network in the critic component guides the neural network in the actor component to achieve action generation by minimizing the loss function, wherein: the neural network in the critic component minimizes the loss function Update, where y t is the temporal difference error, θ is the neural network parameter in the critic component; Q(s t ,a t1 ,a t2 |θ) represents the neural network in the critic component with neural network parameters θ and state s t Perform a joint action (a t1 ,a t2 ), M is the number of samples sampled from the experience replay pool during training, and the target neural network parameters in the critic component are updated by θ′=τθ+(1-τ)θ′, where θ′ represents the target neural network parameters in the critic component and τ represents the soft update parameters;
[0022] The neural network parameters in the actor component are Update, where w t is the neural network parameter in the actor component at time t, w t+1 is the neural network parameter in the actor component at time t+1, l is the learning rate, is the policy gradient, and the policy gradient is obtained by:
[0023] Determine, among which It represents the neural network in the critic component with neural network parameters θ and state s t Perform a joint action in the case of performing a joint action (a t1 ,a t2 )’s value gradient; Indicates that in the actor plus component, the neural network parameter is w and the state is s t The policy action gradient under the condition of ; w is the neural network parameter in the actor-home component, then the target neural network parameter in the actor-home component is updated by w'=τw+(1-τ)w', where w' represents the target neural network parameter in the actor-home component.
[0024] Optionally, the network data stored in the data storage device includes user service data, slice bandwidth data, base station location information, user location information and user mobility data.
[0025] Optionally, after determining that the functional model converges through step S5, the digital twin layer sends the functional model to the physical entity layer, and implements IoT RAN slice switching control and resource allocation based on the switching action and resource allocation action obtained from the functional model.
[0026] Based on the above method, the present invention also proposes a DTN-based Internet of Things RAN slice switching control and resource allocation system, including a DTN-based Internet of Things RAN slice architecture, the DTN-based Internet of Things RAN slice architecture including a physical entity layer and a digital twin layer, the physical entity layer including a variety of Internet of Things user devices and base stations, the digital twin layer including a data storage device, a basic model and a functional model, the basic model is used to monitor the real network in the physical entity layer, and use the collected network data to build a digital twin of the real network; the data storage device is used to store the network data collected by the basic model; the functional model is used to determine the switching action and resource allocation action according to the network operation status in the current basic model and in accordance with the DTN-based Internet of Things RAN slice switching control and resource allocation method described above.
[0027] The remarkable effects of the present invention are:
[0028] (1) The present invention adopts the architecture of IoT RAN slicing based on DTN to pave the way for IoT wireless access network slicing based on digital twin network.
[0029] (2) The present invention proposes a switching control and resource allocation method based on an actor-critic architecture under the architecture of a DTN-based IoT RAN slice. Actor and critic components are designed in this method. First, the actor component not only includes a neural network but also an optimizer. The neural network outputs a switching action. The input and output of the optimizer are the switching and resource allocation actions, respectively. Then, the switching and resource allocation actions are combined as the output of the actor component and the input to the critic component. The critic evaluates the joint decision of switching and resource configuration. Finally, the actor component improves the switching and resource allocation decisions until the performance converges.
[0030] (3) The present invention combines switching costs with system satisfaction to ensure the long-term utility maximization of satisfaction and switching costs in the IoT system. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a DTN-based IoT RAN slicing architecture provided in a specific embodiment of the present invention;
[0032] Figure 2 It is a training framework diagram of a switching control and resource allocation method based on an actor-critic architecture proposed in a specific embodiment of the present invention;
[0033] Figure 3 It is a flow chart of a method in a specific embodiment of the present invention. DETAILED DESCRIPTION
[0034] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.
[0035] like Figure 1 and Figure 2 As shown, this embodiment provides a DTN-based IoT RAN slice switching control and resource allocation method, including the steps of:
[0036] S1. Construct a DTN-based IoT RAN slice architecture, wherein the DTN-based IoT RAN slice architecture includes a physical entity layer and a digital twin layer, and utilize the digital twin layer to monitor and collect the operating status of user devices, network slices, and base stations in the physical entity layer in real time, and trigger a switching instruction when the user satisfaction is lower than a preset threshold or violates the user service quality;
[0037] In specific implementation, the physical entity layer includes a variety of IoT user devices and base stations, and the digital twin layer includes data storage, basic models and functional models, among which:
[0038] The basic model is used to monitor the real network in the physical entity layer and build a digital twin of the real network using the collected network data; including generating digital network topology, digital network elements, RAN slice information and channel status information;
[0039] The data storage device is used to store the network data collected by the basic model, including service requests, location and mobility information of IoT users, available bandwidth resources, minimum rate, maximum allowable delay of network slices, power information and location information of base stations, and channel status, etc.;
[0040] The functional model is used to determine switching actions and resource allocation actions according to the network operation status in the current basic model.
[0041] S2: After receiving the switching instruction, the central controller determines the switching action through the neural network in the actor home component according to the current network status;
[0042] Specifically, after receiving the switching instruction, the central controller, according to the currently observed network status:
[0043]
[0044] Switching actions output by the neural network in the actor component in: Indicates whether user u accesses slice n through base station b at time t-1. If yes, it is 1, otherwise it is 0; W b,n (t) represents the spectrum resources of slice n in base station b at time t; They represent the minimum rate and maximum delay requested by user u at time t respectively; They represent the minimum rate and maximum delay that slice n can provide at time t, respectively. t1 It represents the joint action of selecting base station b and slice n at time t, B is the set of base stations in the physical layer, N is the set of network slices in the physical layer, and U is the set of user equipment in the physical layer.
[0045] S3: using the switching action determined in step S2 as an input of an interior point optimizer in the actor component, and the interior point optimizer outputs a resource allocation action, and the central controller executes the switching action and the resource allocation action as a joint action;
[0046] Specifically, in step S3, the switching action of the neural network output in the actor component Input the interior point optimizer, and the interior point optimizer outputs the resource allocation action represents the spectrum resources allocated to user u by slice n in base station b at time t, and the central controller performs joint action a t =(a t1 ,a t2 ) and enter the next state s t+1 , and get reward r t , and the empirical data (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool.
[0047] S4: input the switching action determined in step S2, the resource allocation action determined in step S3, and the rewards and network parameters formed by the system according to the execution of the joint action into the experience replay pool, use the critic component to evaluate the experience data in the experience replay pool, and input the evaluation results into the actor component for neural network training;
[0048] Specifically, the neural network in the critic component guides the neural network in the actor component to achieve action generation by minimizing the loss function, wherein: Update, where y t is the temporal difference error, θ is the neural network parameter in the critic component; Q(s t ,a t1 ,a t2 |θ) represents the neural network in the critic component when the neural network parameter is θ and the network state is s t Perform a joint action (a t1 ,a t2 ), M is the number of samples sampled from the experience replay pool during training, and is set to 64 in this embodiment, the capacity of the experience replay pool is set to 100000, and the target neural network parameters in the critic component are updated by θ′=τθ+(1-τ)θ′, θ′ represents the target neural network parameters in the critic component, τ represents the soft update parameter, which is set to 0.01 in the specific implementation;
[0049] The neural network parameters in the actor component are Update, where w t is the neural network parameter in the actor component at time t, w t+1 is the neural network parameter in the actor component at time t+1, l is the learning rate, is the policy gradient, and the policy gradient is obtained by:
[0050] Determine, among which It represents the neural network in the critic component with neural network parameters θ and state st Perform a joint action in the case of performing a joint action (a t1 ,a t2 )’s value gradient; Indicates that in the actor plus component, the neural network parameter is w and the state is s t The policy action gradient under the condition of ; w is the neural network parameter in the actor-home component, then the target neural network parameter in the actor-home component is updated by w'=τw+(1-τ)w', where w' represents the target neural network parameter in the actor-home component.
[0051] S5: Repeat steps S2-S4 until the functional model in the digital twin layer maintains convergence.
[0052] Combination Figure 2 It can be seen from the training framework shown that the DTN-based IoT RAN slice switching control and resource allocation method proposed in this embodiment mainly involves a participant component and a critic component in the implementation process. The participant component includes an interior point method optimizer and a neural network, wherein the neural network outputs the switching action, and the interior point method optimization method inputs the resource allocation action respectively. The switching and resource allocation actions are combined as the output of the participant component and the input of the critic component. The critic component evaluates the joint decision of switching and resource allocation, and continuously improves the decision of switching and resource allocation in the participant component. The implementation structure of this method can be divided into three stages, namely, the initialization stage, the experience generation stage, and the update stage.
[0053] Initialization phase: In the initialization phase, the parameters w of the neural network in the actor component and the parameters θ of the critic network, as well as the parameters w′ and θ′, are initialized and the environment state is initialized at the beginning of each round.
[0054] Experience generation phase: The neural network in the actor component generates user switching decisions based on the current state, and then the interior point optimizer generates resource allocation decisions. After the joint action is generated and executed, the reward is received from the digital twin and the next state is entered. The generated experience is stored in the experience replay pool.
[0055] Update phase: The neural networks in the actor component and the critic network randomly extract M mini-batches of samples from the answer buffer as training data to update the neural network parameters offline.
[0056] Combination Figure 3 It can be seen that after the functional model convergence is determined in step S5, the digital twin layer sends the functional model to the physical entity layer, and implements IoT RAN slice switching control and resource allocation based on the switching action and resource allocation action obtained from the functional model.
[0057] according to Figure 3The implementation steps shown in the figure, the whole method can be carried out according to the following process:
[0058] 201. Initialize the system status, and the digital twin layer will collect data in the IoT network.
[0059] 202. Process the collected source data and build a basic model.
[0060] 203. Monitor the user satisfaction of IoT and determine whether it meets user requirements.
[0061] 204. Neural network output switching decision in actor home component.
[0062] The outputs in 205 and 204 are used as inputs to the interior point optimizer in the actor component, and output resource allocation decisions.
[0063] 206. Execute the joint decision of switching and resource allocation in the actor home component.
[0064] 207. Store experience in the experience replay pool.
[0065] 208. The critic component randomly samples experiences from the experience replay pool and minimizes the loss function.
[0066] 209. The actor component neural network randomly samples from the experience replay pool to maximize the policy gradient.
[0067] 210. Update of neural network of actor and critic components.
[0068] 211. Determine whether the functional model of switching and resource allocation has converged.
[0069] 212. The extracted functional model is sent to the physical entity layer.
[0070] 213. The model extracted from step 212 is used for RAN slice switching and resource allocation in the Internet of Things.
[0071] In summary, the present invention provides a DTN-based IoT RAN slice switching control and resource allocation method and system, which can realize fast slice switching and resource allocation of devices in the IoT with the assistance of DTN. DTN monitors the operation of devices and slices in the IoT in real time, and triggers switching once the satisfaction of the access device in the slice is too low. Then the central controller implements switching and resource allocation through an improved reinforcement learning algorithm based on the actor-critic framework. Specifically, in order to ensure maximum user satisfaction and minimum switching cost in the IoT system, the participant component in the present invention includes a neural network for switching control and an optimizer for resource allocation, and then the critic component evaluates the switching and resource allocation actions of the participant component to guide the optimization of the actions in the participant component. The proposed method can effectively improve the long-term utility of device switching in the IoT system with respect to user satisfaction and switching cost.
[0072] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A DTN-based IoT RAN slice switching control and resource allocation method, characterized in that: The following steps are involved: S1. Construct a DTN-based IoT RAN slice architecture, wherein the DTN-based IoT RAN slice architecture includes a physical entity layer and a digital twin layer, and utilize the digital twin layer to monitor and collect the operating status of user devices, network slices, and base stations in the physical entity layer in real time, and trigger a switching instruction when the user satisfaction is lower than a preset threshold or violates the user service quality; S2: After receiving the switching instruction, the central controller determines the switching action through the neural network in the actor home component according to the current network status; S3: using the switching action determined in step S2 as an input of an interior point optimizer in the actor component, and the interior point optimizer outputs a resource allocation action, and the central controller executes the switching action and the resource allocation action as a joint action; S4: input the switching action determined in step S2, the resource allocation action determined in step S3, and the rewards and network parameters formed by the system according to the execution of the joint action into the experience replay pool, use the critic component to evaluate the experience data in the experience replay pool, and input the evaluation results into the actor component for neural network training; S5: Repeat steps S2-S4 until the functional model in the digital twin layer maintains convergence.
2. The DTN-based IoT RAN slice switching control and resource allocation method according to claim 1, characterized in that: The physical entity layer includes a variety of IoT user devices and base stations, and the digital twin layer includes data storage, basic models and functional models, where: The basic model is used to monitor the real network in the physical entity layer and build a digital twin of the real network using the collected network data; The data storage is used to store the network data collected by the basic model; The functional model is used to determine the switching action and the resource allocation action according to the network operation status in the current basic model.
3. The DTN-based IoT RAN slice switching control and resource allocation method according to claim 1 or 2, characterized in that: In step S2, after receiving the switching instruction, the central controller performs the following operations according to the currently observed network status: Switching actions output by the neural network in the actor component in: Indicates whether user u accesses slice n through base station b at time t-1. If yes, it is 1, otherwise it is 0; W b,n (t) represents the spectrum resources of slice n in base station b at time t; They represent the minimum rate and maximum delay requested by user u at time t respectively; They represent the minimum rate and maximum delay that slice n can provide at time t, respectively. t1 It represents the joint action of selecting base station b and slice n at time t, B is the set of base stations in the physical layer, N is the set of network slices in the physical layer, and U is the set of user equipment in the physical layer.
4. The DTN-based IoT RAN slice switching control and resource allocation method according to claim 3 is characterized in that: Step S3 switches the output of the neural network in the actor component Input the interior point optimizer, and the interior point optimizer outputs the resource allocation action represents the spectrum resources allocated to user u by slice n in base station b at time t, and the central controller performs joint action a t =(a t1 ,a t2 ) and enter the next state s t+1 , and get reward r t , and the empirical data (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool.
5. The DTN-based IoT RAN slice switching control and resource allocation method according to claim 1 or 4, characterized in that: In step S4, the neural network in the critic component guides the neural network in the actor component to achieve action generation by minimizing the loss function, wherein: the neural network in the critic component minimizes the loss function Update, where y t is the temporal difference error, θ is the neural network parameter in the critic component; Q(s t ,a t1 ,a t2 |θ) represents the neural network in the critic component with neural network parameters θ and state s t Perform a joint action (a t1 ,a t2 ), M is the number of samples sampled from the experience replay pool during training, and the target neural network parameters in the critic component are updated by θ′=τθ+(1-τ)θ′, where θ′ represents the target neural network parameters in the critic component and τ represents the soft update parameters; The neural network parameters in the actor component are Update, where w t is the neural network parameter in the actor component at time t, w t+1 is the neural network parameter in the actor component at time t+1, l is the learning rate, is the policy gradient, and the policy gradient is obtained by: Determine, among which It represents the neural network in the critic component with neural network parameters θ and state s t Perform a joint action in the case of performing a joint action (a t1 ,a t2 )’s value gradient; Indicates that in the actor plus component, the neural network parameter is w and the state is s t The policy action gradient under the condition of ; w is the neural network parameter in the actor-home component, then the target neural network parameter in the actor-home component is updated by w'=τw+(1-τ)w', where w' represents the target neural network parameter in the actor-home component.
6. The DTN-based IoT RAN slice switching control and resource allocation method according to claim 2, characterized in that: The network data stored in the data storage device includes user service data, slice bandwidth data, base station location information, user location information and user mobility data.
7. The DTN-based IoT RAN slice switching control and resource allocation method according to claim 1, characterized in that: After determining that the functional model converges through step S5, the digital twin layer sends the functional model to the physical entity layer, and implements IoT RAN slice switching control and resource allocation based on the switching action and resource allocation action obtained from the functional model.
8. A DTN-based IoT RAN slice switching control and resource allocation system, characterized in that: The invention comprises a DTN-based IoT RAN slicing architecture, wherein the DTN-based IoT RAN slicing architecture comprises a physical entity layer and a digital twin layer, wherein the physical entity layer comprises a variety of IoT user devices and base stations, and the digital twin layer comprises a data storage device, a basic model and a functional model, wherein the basic model is used to monitor the real network in the physical entity layer and construct a digital twin of the real network using the collected network data; and the data storage device is used to store the network data collected by the basic model; The functional model is used to determine the switching action and resource allocation action according to the network operation status in the current basic model and in accordance with the DTN-based Internet of Things RAN slice switching control and resource allocation method as described in any one of claims 1-7.
