Resource allocation method for digital twin auxiliary end-to-end deterministic time delay network slices in industrial Internet of Things
Through the end-to-end network slice resource allocation method assisted by digital twins, combined with random network calculations and multi-agent deep reinforcement learning, resource allocation is optimized, and the business needs of high latency and reliability requirements in the industrial Internet of Things are solved, and the determination is achieved maximization of sexual delay and system utility.
Patent Information
- Application Number
- CN202410988743.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-23
AI Technical Summary
The existing technology is difficult to meet the business needs of high latency and reliability requirements in the industrial Internet of Things. The existing research only considers minimizing network latency from a single-sided network domain or maximizing network resource utilization, and cannot meet end-to-end network slicing. deterministic delay performance requirements.
Using the end-to-end network slice resource allocation method with digital twin assist, the end-to-end network slice architecture with digital twin assist is optimized by using random network calculations and multi-agent deep reinforcement learning algorithms. Multi-dimensional resource allocation, proposed a multi-agent-priority experience replay of flexible actor-criticist algorithm (DT-PER-MASAC) with digital twin assisted to maximize system utility and ensure deterministic delay.
It realizes the improvement of system effectiveness while meeting the deterministic delay requirements, solves the problem of cross-network domain resource allocation caused by dynamic changes in the status of IoT terminal service requests and service nodes, and improves the efficient utilization of network resources.
Smart Images

Figure CN119031484A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of mobile communication technology and relates to a resource allocation method for digital twin-assisted end-to-end deterministic delay network slicing in an industrial Internet of Things. Background Art
[0002] With the development of IoT technology, a large number of IoT devices are connected to the network. The existing network cannot fully meet the strict and diverse requirements of latency-sensitive services for latency, reliability, etc. In order to meet the Quality of Service (QoS) requirements of transmission services, network service providers (NSP) use network slicing technology to abstract the physical resources provided by infrastructure providers (InP) into virtual resources. NSP can configure virtual resources according to user needs and provide personalized services. As a key technology in 5G networks, network slicing usually requires resources from multiple network domains. There is a very complex trade-off between these resources and slice performance. Therefore, resource allocation is crucial to the resource utilization and network performance of network slices.
[0003] Due to the dynamic changes in business needs and device status information, it is currently difficult to ensure business QoS under information uncertainty. Meeting strict and diverse business needs has become an important and urgent goal of network deployment. In order to achieve this goal, it is necessary to combine new technologies to obtain real-time information of the network. The 6G era will be a digital era. Digital Twin (DT), as one of the key technologies of 6G, combined with communication networks, maps entities in the physical network (including network elements, user devices, etc.) to virtual space, and can build a Digital Twin Network (DTN) that is consistent with the physical network. In the industrial Internet of Things scenario, by integrating the Internet of Things and DT that support Mobile Edge Computing (MEC), real-time monitoring of the status of network elements in the entire end-to-end network slice can be achieved, and perception data and decisions can be provided through functional models, bringing more intelligent and efficient solutions to industrial applications.
[0004] The diversification of business needs has put forward higher requirements for network flexibility. Although existing technologies have studied aspects such as DT-assisted network slicing and resource allocation, the dynamic changes in business requests and service node status of IoT terminals make it difficult for cross-network domain resource allocation in end-to-end network slicing to meet latency and reliability requirements. Existing research only considers minimizing network latency or maximizing network resource utilization from a single-sided network domain, which cannot meet new business needs with high latency and reliability requirements. A key issue in providing deterministic services is how to allocate the right amount of resources to ensure QoS requirements within the network slice. Deterministic latency performance rather than latency minimization can meet the network performance requirements of time-critical services and should become a new direction for network resource optimization. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a resource allocation method for digital twin-assisted end-to-end deterministic latency network slicing in the industrial Internet of Things, which can maximize the system utility while ensuring deterministic latency.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] A resource allocation method for digital twin-assisted end-to-end deterministic latency network slicing in industrial Internet of Things, the method comprising the following steps:
[0008] S1: In the industrial IoT scenario, build an end-to-end network slicing architecture assisted by digital twin DT, and use digital twin to synchronize VNF to perceive each network slice request information, including the source node, destination node, base station location information, virtual link set, node resource information, latency requirements and reliability requirements of the service flow;
[0009] S2: Consider the impact of computing resource offset and synchronization delay in the digital twin mapping process on the end-to-end delay of the service;
[0010] S3: Stochastic network calculus (SNC) is introduced to characterize the upper bound of delay in service transmission, and the service arrival model and service model statistically analyzed by the digital twin model are used to characterize the relationship between the upper bound of delay and reliability.
[0011] S4: Jointly optimize the multi-dimensional resource allocation problem of time-frequency, computing, storage and bandwidth with the goal of maximizing system utility;
[0012] S5: To solve the optimization problem, a digital twin-assisted multi-agent priority experience replay flexible actor-critic algorithm DT-PER-MASAC is proposed to learn the network slicing deployment strategy that maximizes the system utility while ensuring the latency and reliability requirements of end-to-end network slicing.
[0013] Further, the DT-assisted end-to-end network slicing architecture includes a physical network layer, a digital twin layer, and an application service layer;
[0014] The physical network layer includes physical network elements, namely base stations (BS) in the radio access network (RAN) domain, mobile edge computing MEC servers close to the BS, IoT terminals, and multiple servers in the core network (CN) domain. The physical network elements support both software and virtualization technologies to orchestrate and manage all network functions in the network.
[0015] The digital twin layer is composed of RAN domain and CN domain mapping, including multiple virtual devices, virtual nodes and links. The digital twin layer assists the end-to-end network, predicts and analyzes the transmission status based on real-time data and historical data, and further adjusts the resource allocation plan in the digital twin space;
[0016] The application service layer checks whether the service level agreement of the deployed end-to-end slices meets the requirements by interacting with the DT layer.
[0017] Furthermore, the network resources of the end-to-end network slice include heterogeneous communication resources, computing resources, storage resources and link bandwidth resources.
[0018] Furthermore, the service is provided by RAN slices. The delay model of RAN includes uplink transmission delay and processing delay. CN slices consider propagation delay, transmission delay, processing delay and DT synchronization delay. The propagation delay and transmission delay are calculated in the routing stage, and the processing delay and DT synchronization delay are calculated at the node.
[0019] Furthermore, in S3, the arrival process of the service data packets in the network domain and the service process of the service node are analyzed by using SNC and the moment generating function to obtain the upper bound of the delay violation probability, and to establish a connection between the delay and the reliability of service transmission under the given arrival traffic distribution and delay limit:
[0020] P[τ m,E2E <T m ]≥1-ε m,max
[0021] In the formula, τ m,E2E is the total end-to-end actual delay of service m in DTN, including RAN delay τ m,RAN and CN delay τ m,CN , denoted as τ m,E2E =τ m,RAN +τ m,CN , T m Indicates the end-to-end delay requirement of service m data packets, p m =1-εm,max Represents the service transmission reliability requirement, ε m,max Indicates the maximum transmission violation probability of service delay.
[0022] In the formula, the total end-to-end actual delay of service m in DTN is expressed as τ m,E2E =τ m,RAN +τ m,CN , T m Indicates the end-to-end delay requirement of service m data packet, p m =1-ε m,max Indicates the service transmission reliability requirement.
[0023] Furthermore, in S4, the problem of ensuring the determinism of the DT-assisted E2E network slice delay while maximizing the total utility of the system is expressed as:
[0024]
[0025] Where N RB Indicates the time and frequency resources allocated in the RAN domain. Indicates the computing resources on the server nodes in the RAN domain and CN domain, Indicates storage resources in the CN domain. represents the bandwidth resources between nodes, M represents the total number of services, Represents the total utility of the system.
[0026] Furthermore, in S5, the optimization problem is transformed into a Markov decision process model, including states, actions, transition probabilities and rewards, which can be expressed as This includes the state set Action Set State transition probability set Bonus Set The agent initiates a service request for each IoT terminal in the system scenario, and each agent selects an action from the action space by considering the network status.
[0027] Further, in S5, the DT-PER-MASAC algorithm comprises the following steps:
[0028] S81: Initialize the actor network parameters of the agent, the evaluation network parameters and target network parameters in the critic network, and the experience replay pool;
[0029] S82: Determine whether the number of iterations exceeds the set number. If yes, stop the iteration. Otherwise, continue to execute S83.
[0030] S83: For N samples in the experience replay pool, obtain the initial local state information of each agent from the DT-assisted E2E network slicing environment;
[0031] S84: Determine whether the set number of training times is exceeded, if yes, stop the training, otherwise continue to execute S85;
[0032] S85: Update the actor network and critic network according to the priority experience playback mechanism;
[0033] S86: Recalculate the temporal difference-error (TD-error) value of the extracted experience, update the priority and temperature coefficient of the experience, and soft-update the target network parameters.
[0034] 9. The resource allocation method for digital twin-assisted end-to-end deterministic delay network slices in the industrial Internet of Things according to claim 7 is characterized in that: the Prioritized Experience Replay (PER) in the DT-PER-MASAC algorithm uses priority to calculate the probability of random selection of the extraction, and the priority of sample i is expressed as:
[0035]
[0036] Among them, s t and t+1 are the states of time slot t and the next time slot, a t and a t+1 are the actions selected in time slot t and the next time slot, r(s t ,a t ) is based on the state s t Execute action a t Rewards received, and are the action value functions of the current time slot and the next time slot, α is the temperature coefficient, and π φ (a t+1 |s t+1 ) indicates that the strategy is in state s t+1 Take action a t+1 The probability of is a positive constant that approaches 0 infinitely, ensuring that every transition can be sampled even if the TD error is zero.
[0037] The beneficial effects of the present invention are as follows: In view of the problem that the dynamic changes in the business requests and service node states of IoT terminals cause the cross-network domain resource allocation in the end-to-end network slicing to be difficult to meet the latency and reliability requirements, the present invention proposes a digital twin-assisted end-to-end deterministic latency network slicing resource allocation method. First, a DT-assisted end-to-end network slicing frame is constructed. Secondly, based on the random network calculus theory, the probability of end-to-end latency violations is analyzed, and a DT-assisted end-to-end resource allocation model is established to ensure deterministic latency while maximizing system utility. Finally, it is proposed to use a multi-agent deep reinforcement learning algorithm in a distributed architecture to solve complex optimization problems and achieve efficient network resource allocation. The present invention can improve system utility while meeting deterministic latency requirements.
[0038] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0040] Figure 1 Schematic diagram of end-to-end network slicing architecture assisted by digital twin;
[0041] Figure 2 This is the network structure diagram of the DT PER-MASAC algorithm. DETAILED DESCRIPTION
[0042] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0043] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.
[0044] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0045] An embodiment of the present invention proposes a resource allocation method for digital twin-assisted end-to-end deterministic latency network slicing in the industrial Internet of Things, which can maximize system utility while ensuring deterministic latency.
[0046] The steps of this method are as follows:
[0047] S1: In the industrial IoT scenario, build an end-to-end network slicing architecture assisted by digital twins, and use digital twins to synchronize VNFs to perceive each network slice request information, including the source node, destination node, base station location information, virtual link set, node resource information, latency requirements, and reliability requirements of the service flow; Figure 1 Schematic diagram of end-to-end network slicing architecture assisted by digital twins.
[0048] The physical network layer includes physical network elements, such as BS in the RAN domain, MEC servers close to the BS, IoT terminal devices, and multiple servers in the CN domain. Each server contains one or more virtual machines, and the servers are connected by optical fibers. In the physical network layer, all physical network elements support both software and virtualization technologies to effectively orchestrate and manage all network functions, network resources, and network traffic data in the network.
[0049] The digital twin function is deployed on multiple CN servers, and determines the resource allocation strategy for the physical network by updating the wireless transmission channel time and frequency resources, server resource status, service request status, historical data, etc. in real time. The digital twin layer is composed of RAN domain and CN domain mapping, including multiple virtual IoT terminal devices, a MEC server and multiple CN domain VNFs, more specifically including the location information of the service terminal device, service delay requirements and guaranteed transmission probability, link topology between VNF network elements, and bandwidth resources. The service m of the digital twin layer is represented as
[0050]
[0051] in, Represents the digital twin of the IoT terminal m, including the terminal's location coordinate information and service SLA (latency, reliability). Represents the collection of MEC servers in the RAN domain and servers in the CN domain, including the usage of the server's computing resources and storage resources. The business digital twin is constructed by extracting the operating status of multiple related devices.
[0052] In the framework of DT-assisted end-to-end network slicing, the physical network layer and the digital twin layer continuously interact to realize data collection and slice control management. By mapping the characteristics and states of physical entities and communication environments, a more accurate real-time DT model can be built.
[0053] Building a fully mapped twin model of all physical network elements will consume a lot of network resources. Based on the proposed digital twin network architecture, the digital twin synchronization and resource allocation process is as follows:
[0054] First, the raw data is obtained from the physical network layer, such as device location, base station location, server available resources, service quality indicators, etc. After the raw data is collected, necessary preprocessing is performed, such as data cleaning, to ensure the effectiveness and accuracy of further analysis.
[0055] Secondly, the DT VNF collects synchronous data packets consisting of key information at a certain sampling rate and calculates the synchronization delay of transmitting synchronous data. It must ensure that the transmission of data packets remains within the maximum allowed synchronization delay to maintain the real-time performance of the DT model.
[0056] Then, the DT VNF stores and analyzes the received data, and builds or updates the DT model to reflect the real-time status of the physical network. Furthermore, in the constructed DT model, the AI algorithm model is used to formulate or adjust the resource allocation plan for network slices according to the slice requirements of the business and the current resource situation, and the plan is distributed to the virtual network for verification. The virtual network will feed back the obtained synchronization delay to the DT model to decide whether to perform synchronization processing. If it is observed that the synchronization delay obtained in the virtual network is greater than the maximum synchronization delay, it may be that the data or parameters collected in the DT model have large deviations, and the DT mapping model needs to be adjusted. Repeat and update this process until the synchronization mapping requirements are met.
[0057] Finally, the resource allocation plan is sent to the physical network layer to guide the configuration and adjustment in the physical network. The physical network layer and the digital twin layer continuously exchange information to form a closed-loop control system, thereby achieving highly adaptive and optimized network management.
[0058] The computing power is increased in the computer room close to the access network, and the RAN virtualization function is realized through the MEC server. The virtual network function is instantiated by allocating physical resources (CPU, RAM). In the single-cell network multi-service MEC system, a single BS has 1 high-performance MEC server and contains service requests of m IoT terminal devices in the range. The service set M = {1, 2, ..., m}, each service m∈M, is transmitted using the OFDMA protocol, and the service terminal transmits data packets to the BS through the uplink.
[0059] Each service is served by one RAN slice, and the total bandwidth of the RAN slice is B = J·B J , the frequency domain is divided into J sub-channels with a bandwidth of B J In the time domain, the scheduling frame can be divided into K scheduling subframes of length Δt. The minimum granularity resource block (RB) set of time-frequency resources is R = {1, 2, …, R}, and the total number of RBs is R = J·K.
[0060] A CN network slice is a virtual link connected by multiple VNF nodes. Different virtual links have different latency requirements, transmission probabilities, and resource requirements. CN is represented as an undirected graph. represents the node set, E={1,2,…,e,…,E} represents the link set between nodes. Each server node n deploys one or more virtual machines, denoted as V n ={1,2,…,b}, where b represents the number of virtual machines hosted by node n. Assuming that each virtual machine is dedicated to a VNF and that VNFs cannot share the same virtual machine resources, the set of VNFs on node n is represented as The total number of virtual machines in CN is
[0061] Different services are run by different slices, and the network service provider allocates appropriate network resources to ensure deterministic latency, including computing resources, storage resources for VNFs on transmission links, and bandwidth resources for routing traffic between VNFs. The total resource capacity of each node n∈N is expressed as in are the total capacity of computing resources and storage resources respectively. There is limited bandwidth capacity between two nodes i and j represent any two nodes in the node set N, and i is not equal to j. Different virtual links will not share the same VNF, and each VNF uses exclusive network resources.
[0062] S2: Consider the impact of computing resource offset and synchronization delay in the digital twin mapping process on the end-to-end delay of the service;
[0063] Whether the RB resource r on the BS is allocated to service m is represented by a binary variable Indicates that a resource block can only be used for one service transmission:
[0064]
[0065] The number of RBs required for transmission of service m is The amount of time and frequency resources used by all services cannot exceed the total amount of time and frequency resources:
[0066]
[0067] Although the DT model represents the operating state of the real network as accurately as possible, there are still mapping errors due to the limitations of the DT modeling method and modeling data acquisition. Consider the digital twin mapping process with a computing resource offset Δf m , whose range can be obtained by induction based on the long-term operation of DT, |Δf m |≤Δf max , Δf max is the maximum computing resource offset. When the computing resource offset exceeds the threshold Δf max , it indicates that the current twin network element computing resource mapping error is large, and the computing resource offset needs to be corrected, that is, the DT mapping model needs to be synchronized. The computing resources actually allocated to service m are ψ CPU,m =ψ CPU,m +Δf m ,ψ CPU,m ≥0, the allocated computing resources cannot exceed the total computing resources of the MEC server:
[0068]
[0069] The RAN delay model includes uplink transmission delay and processing delay. The BS implements RAN slicing by allocating RBs. The total RAN slice delay of service m is expressed as τ m,RAN =t up,m +t comp,m .
[0070] The CN slice model of the transmission service m in the digital twin layer is represented as a virtual undirected graph in are the virtual nodes and virtual links of service m respectively. Contains parameters They respectively represent the source node, destination node, delay requirement and reliability requirement of the virtual link transmitting service m.
[0071] The physical network model is mapped to the digital twin layer, setting binary variables Indicates the mapping of physical network node n to VNFu, a binary variable Indicates that the physical link (i, j) is mapped to the virtual link (u, v). Each physical node can only be mapped to one VNF node in the same virtual link, which is represented by
[0072] The computing resources and storage resources of each node are shared by multiple VNFs deployed on the node. Each VNF instance has exclusive resources and can only belong to a certain service link. The computing resources and storage resources of node n are constrained as follows:
[0073]
[0074] in, Indicates the computing resources allocated to VNF u for service m, in CPU cycles. Indicates the storage resources allocated to VNF u.
[0075] The bandwidth resources allocated between VNF u and v on the virtual link in the transmission service m are The bandwidth resources allocated to the virtual link should be smaller than the actual bandwidth resources between physical links. The link bandwidth resource constraints are:
[0076]
[0077] A virtual link is dedicated to transmitting one type of service. The slices monopolize the acquired resources, and the link will not be congested. Therefore, there is no queuing for service transmission, and queuing delay is not considered. The CN domain in the digital twin layer considers propagation delay, transmission delay, processing delay, and synchronization delay required for DTN synchronization. The propagation delay and transmission delay are calculated in the routing stage, and the processing delay and synchronization delay are calculated at the node. The actual delay in the CN domain is expressed as:
[0078] τ m,CN =t prop,m +t trans,m +t proc,m +t syn,m
[0079] The node processing delay can be regarded as a function of the computing resources and storage resources allocated to VNF u on the physical node n. Considering the offset of computing resources between the twin network element and the real network element, the processing delay of VNF u on node n is expressed as:
[0080]
[0081] The DT model requires continuous synchronous data transmission with the modeled object to maintain consistency, and the size of the synchronization data packet is relatively small in actual transmission. Resource management and scheduling require real-time information updates. If the node synchronization delay is large, it will lead to untimely resource scheduling, thereby increasing end-to-end delay. However, data packet loss may occur in the communication between the DT server and the physical network element nodes. In order to measure the state consistency between the DTN and its physical network elements, the DT synchronization delay is introduced. The DT synchronization delay is the maximum synchronous data transmission delay between the DT model and the mapped server node set:
[0082]
[0083] in, It represents the time required to send one unit of data per unit distance, D syn,m Indicates the size of the synchronization data packet, d i,u Represents the distance between DT synchronization node i and VNF u. The maximum synchronization delay defines the maximum allowable time for data to be transmitted from the physical network layer to the digital twin layer to complete synchronization. By setting the maximum synchronization delay, it can be ensured that the DT model can reflect the current status of the physical network in a timely manner. The DT synchronization delay should be less than the maximum synchronization delay, which is expressed as t syn,m ≤t syn,max .
[0084] The total end-to-end actual delay of service m in DTN is τ m,E2E =τ m,RAN +τ m,CN , the end-to-end delay constraint of service m is:
[0085] τ m,E2E ≤T m
[0086] S3: Introduce random network calculus to characterize the upper bound of latency in service transmission, and use the service arrival model and service model statistically analyzed by the digital twin model to characterize the relationship between the upper bound of latency and reliability;
[0087] SNC is a theoretical tool for network performance analysis. It can guarantee the network's delay boundary performance within a certain probability range. It is very suitable for service transmission with strict delay requirements and taking into account resource utilization. SNC uses the arrival curve and service curve to constrain the arrival process of service data and the service process of service nodes, and can further analyze the backlog limit and delay limit. The backlog limit is the maximum vertical distance between the two curves, and the delay limit is the maximum horizontal distance between the two curves. For service m, all data packets have the same delay requirement T m , ε m,maxis the maximum delay violation probability of service m. If the service data packet has a first-in-first-out (FIFO) policy, it is guaranteed that the service m data packet meets the end-to-end delay requirement T m , reliability requirement p m =1-ε m,max The following transmission is represented as
[0088] P[τ m,E2E <T m ]≥1-ε m,max
[0089] In the service flow arrival model, service flows are kept independent and identically distributed. The process of service data packets from IoT device terminals to BS is a Poisson process. The arrival process of service m arriving during the time [τ, t] is:
[0090]
[0091] Among them, a m (i) represents the data packet size of service m from the service terminal to the BS in the i-th time slot. Given a constant data packet size D m , the cumulative arrival process is A(t)=D m N(t), calculate A m The moment generating function of (τ,t) is
[0092]
[0093] Among them, the free parameter θ>0, for the SNC problem, the affine reaches the envelope upper bound
[0094]
[0095] Using affine to reach envelope parameters and Restricted Arrival Process A m (τ,t):
[0096]
[0097] The wireless channel is modeled as a Rayleigh fading channel, and the instantaneous signal-to-noise ratio of service m transmitted in time slot i is recorded as The probability density function of g follows an exponential distribution with a mean of 1. If the channel gain of each time slot is independent and identically distributed, then the service process is also independent and identically distributed, and the channel transmission service process is The moment generating function is:
[0098]
[0099] make Get the channel transmission service process moment generating function:
[0100]
[0101] Get the affine service envelope parameters:
[0102]
[0103] Service data is transmitted to CN via BS, and data packets are processed by multiple server nodes. Server node i is modeled as having a total computing capacity of C i ,φ m ∈[0,1] represents the resource allocation weight factor for business m. The service calculation process of server node i is: The moment generating function of the server computing service is get
[0104] The service process S(t) of the business data packet is represented by two parts: the wireless transmission service process and the server computing service process. According to the service cascade theorem, The service node i∈(1,n) is calculated, and n represents the number of service nodes on the transmission link of service m. Let the dynamic server S0 represent the service of the wireless channel, and the dynamic servers S1,…,S n represents the service of the server on the transmission link, the free parameter ∈>0 and
[0105] When each service process is independent of each other, the end-to-end network service process constraint can be obtained as follows:
[0106]
[0107] Under the condition that the arrival process and the service process are independent, the upper bound of the delay violation probability is obtained:
[0108]
[0109] Different service transmissions have different reliability requirements. The reliability requirement of service m is expressed as the relationship between the upper bound of the end-to-end delay violation probability and the maximum transmission violation probability of service delay:
[0110] ε m <ε m,max
[0111] S4: Jointly optimize the multi-dimensional resource allocation problem of time-frequency, computing, storage and bandwidth with the goal of maximizing system utility;
[0112] NSP accepts the request for slice m and allocates data rate R m Generate income, represents the data rate unit price of slice m, and the revenue function is expressed as:
[0113]
[0114] use They represent the unit cost of allocating time-frequency resource blocks in the RAN domain and computing resources in the MEC server for slice m, respectively. The cost function of slice m in the RAN domain is:
[0115]
[0116] represents the unit cost of computing resources of nodes on slice m in the CN domain, Represents the unit cost of storage resources of the node, represents the unit cost of bandwidth resources on the transmission link. The cost function of slice m in the CN domain is:
[0117]
[0118] The total cost function of DT-assisted E2E network slicing is expressed as The overall utility function is expressed as:
[0119]
[0120] Among them, Θ1 and Θ2 are used to balance and scale the revenue and cost of different service types respectively. In order to ensure the determinism of the DT-assisted E2E network slicing latency while maximizing the total utility of the entire system, the optimization objective is expressed as:
[0121]
[0122] S5: To solve the optimization problem, a digital twin-assisted multi-agent priority experience replay flexible actor-critic algorithm DT-PER-MASAC is proposed to learn the network slicing deployment strategy that maximizes the system utility while ensuring the latency and reliability requirements of end-to-end network slicing. Figure 2 This is the network structure diagram of the DT PER-MASAC algorithm.
[0123] The optimization problem is transformed into a Markov decision process model, including states, actions, transition probabilities, and rewards, expressed as This includes the state set Action Set State transition probability set Bonus Set The agent initiates a service request for each IoT terminal in the system scenario, and each agent selects an action from the action space by considering the network status;
[0124] The Soft Actor-Critic (SAC) algorithm is an offline policy algorithm based on a continuous action space and oriented to the maximum entropy RL framework. The optimal strategy of this method is to maximize its entropy regularized reward, making the algorithm more stable. In addition to learning the strategy to maximize the expected cumulative reward, the SAC algorithm objective function also requires that the action entropy output by each strategy is maximized:
[0125]
[0126] Where T is the number of time steps that the agent interacts with the environment in each round, and ρπ represents the strategy π(a t |s t ) under the trajectory distribution, α is the hyperparameter of the temperature coefficient, which is used to determine the importance of entropy.
[0127] Update the actor network according to minimizing the KL divergence, in order to minimize the optimization target J using the stochastic gradient method π (Θ), the sampling process is moved out of the computational graph by re-parameterization, and the state and random noise are used as inputs. t =f θ (ε t ,s t ), noise ε t Obey the standard normal distribution. We get:
[0128]
[0129] To automatically update α, represents the hyperparameter of the target entropy, and the optimized loss function is:
[0130]
[0131] SAC uses two Q networks and selects the smaller Q value as the target Q value each time to ensure fast and stable training, and updates the critic network by minimizing the Bellman error.
[0132] The SAC algorithm is only applicable to a single agent. If the overall model is built and centralized single-agent reinforcement learning is used, there is a problem of a large action space and a large signaling overhead. In addition, if completely independent reinforcement learning is used for each agent, when a certain agent is being discussed, the strategies of other agents may change at any time, resulting in an unstable environment. The SAC algorithm is applied to a multi-agent framework with centralized training and distributed execution using a multi-agent method, and the optimization problem is solved through collaboration between agents.
[0133] The experience replay buffer generally uses uniform sampling to sample data. This random update will use a large amount of data with little or no effect. At the same time, since each update uses uniform sampling to extract sample data, the algorithm converges slowly, and some rare but important data is not extracted during the entire algorithm update process.
[0134] Using the PER structure, all data are sorted by priority (time difference TD error). Data with high priority, that is, data with large TD error, has a greater probability of being selected. Therefore, compared with the general Replay Buffer, PER has a greater chance of selecting valid data in each time slot of the algorithm update. The algorithm update gradient is larger and the algorithm converges faster. PER uses priority to calculate the probability of random selection of the extraction. The priority of sample i is expressed as:
[0135]
[0136] Among them, s t and t+1 are the states of time slot t and the next time slot, a t and a t+1 are the actions selected in time slot t and the next time slot, r(s t ,a t ) is based on the state s t Execute action a t Rewards received, and are the action value functions of the current time slot and the next time slot, α is the temperature coefficient, and π φ (a t+1 |s t+1 ) indicates that the strategy is in state s t+1 Take action a t+1 The probability of is a positive constant that approaches 0 infinitely, ensuring that every transition can be sampled even if the TD error is zero.
[0137] The probability of sampling i is defined as κ is a sampling preference control parameter, which is used to determine the sampling preference between uniform sampling and greedy sampling. When κ=0, it is uniform sampling, and when κ=1, it is greedy sampling.
[0138] The priority distribution in the experience replay pool is constantly changing, and the correlation between samples may change the results of algorithm convergence.
[0139] In order to ensure that there is no correlation between samples, the importance sampling method is used. The importance sampling weight of sample i is defined as:
[0140]
[0141] Where ζ is the importance weight control parameter, which is used to determine the extent to which the impact of the priority replay mechanism on the convergence result is offset.
[0142] Calculate the experience extraction probability P(i) and the importance sampling weight w i After that, the loss function of updating the Critic network becomes:
[0143]
[0144] The DT-PER-MASAC algorithm specifically includes the following steps:
[0145] S81: Initialize the actor network parameters of the agent, the evaluation network parameters and target network parameters in the critic network, and the experience replay pool;
[0146] S82: Determine whether the number of iterations exceeds the set number. If yes, stop the iteration. Otherwise, continue to execute S83.
[0147] S83: For N samples in the experience replay pool, obtain the initial local state information of each agent from the DT-assisted E2E network slicing environment;
[0148] S84: Determine whether the set number of training times is exceeded, if yes, stop the training, otherwise continue to execute S85;
[0149] S85: Update the actor network and critic network according to the priority experience playback mechanism;
[0150] S86: Recalculate the TD-error value of the extracted experience, update the priority and temperature coefficient of the experience, and soft-update the target network parameters.
[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A resource allocation method for end-to-end deterministic latency network slicing assisted by digital twins in industrial Internet of Things, characterized by: The method comprises the following steps: S1: In the industrial IoT scenario, build an end-to-end network slicing architecture assisted by digital twin DT, and use digital twin to synchronize VNF to perceive each network slice request information, including the source node, destination node, base station location information, virtual link set, node resource information, latency requirements and reliability requirements of the service flow; S2: Consider the impact of computing resource offset and synchronization delay in the digital twin mapping process on the end-to-end delay of the service; S3: Stochastic network calculus (SNC) is introduced to characterize the upper bound of delay in service transmission, and the service arrival model and service model statistically analyzed by the digital twin model are used to characterize the relationship between the upper bound of delay and reliability. S4: Jointly optimize the multi-dimensional resource allocation problem of time-frequency, computing, storage and bandwidth with the goal of maximizing system utility; S5: To solve the optimization problem, a digital twin-assisted multi-agent priority experience replay flexible actor-critic algorithm DT-PER-MASAC is proposed to learn the network slicing deployment strategy that maximizes the system utility while ensuring the latency and reliability requirements of end-to-end network slicing.
2. The resource allocation method for digital twin-assisted end-to-end deterministic latency network slices in the industrial Internet of Things according to claim 1 is characterized in that: The DT-assisted end-to-end network slicing architecture includes a physical network layer, a digital twin layer, and an application service layer; The physical network layer includes physical network elements, namely, base stations BS in the radio access network RAN domain, mobile edge computing MEC servers close to the BS, IoT terminals, and multiple servers in the core network CN domain. The physical network elements support both software and virtualization technologies to orchestrate and manage all network functions in the network. The digital twin layer is composed of RAN domain and CN domain mapping, including multiple virtual devices, virtual nodes and links. The digital twin layer assists the end-to-end network, predicts and analyzes the transmission status based on real-time data and historical data, and further adjusts the resource allocation plan in the digital twin space; The application service layer checks whether the service level agreement of the deployed end-to-end slices meets the requirements by interacting with the DT layer.
3. The resource allocation method for digital twin-assisted end-to-end deterministic latency network slices in the industrial Internet of Things according to claim 2 is characterized in that: The network resources of the end-to-end network slice include heterogeneous communication resources, computing resources, storage resources and link bandwidth resources.
4. The resource allocation method for digital twin-assisted end-to-end deterministic latency network slices in the industrial Internet of Things according to claim 1 is characterized in that: The service is provided by RAN slices. The delay model of RAN includes uplink transmission delay and processing delay. CN slices consider propagation delay, transmission delay, processing delay and DT synchronization delay. The propagation delay and transmission delay are calculated in the routing stage, and the processing delay and DT synchronization delay are calculated at the node.
5. The resource allocation method for digital twin-assisted end-to-end deterministic latency network slices in the industrial Internet of Things according to claim 1 is characterized in that: In S3, SNC and moment generating function are used to analyze the arrival process of service data packets in the network domain and the service process of service nodes, obtain the upper bound of delay violation probability, and establish a connection between delay and service transmission reliability under the condition of given arrival traffic distribution and delay limit: P[t m,E2E <T m ]≥1-e m,max In the formula, τ m,E2E is the total end-to-end actual delay of service m in DTN, including RAN delay τ m,RAN and CN delay τ m,CN , denoted as τ m,E2E =τ m,RAN +τ m,CN , T m Indicates the end-to-end delay requirement of service m data packets, p m =1-ε m,max Represents the service transmission reliability requirement, ε m,max Indicates the maximum transmission violation probability of service delay.
6. The resource allocation method for digital twin-assisted end-to-end deterministic latency network slices in the industrial Internet of Things according to claim 1 is characterized in that: In S4, the problem of ensuring the determinism of the DT-assisted E2E network slice latency while maximizing the total utility of the system is expressed as: Where N RB Indicates the time and frequency resources allocated in the RAN domain. Indicates the computing resources on the server nodes in the RAN domain and CN domain, Indicates storage resources in the CN domain. represents the bandwidth resources between nodes, M represents the total number of services, Represents the total utility of the system.
7. The resource allocation method for digital twin-assisted end-to-end deterministic latency network slices in the industrial Internet of Things according to claim 1 is characterized in that: In S5, the optimization problem is transformed into a Markov decision process model, including states, actions, transition probabilities and rewards, which can be expressed as This includes the state set Action Set State transition probability set Bonus Set The agent initiates a service request for each IoT terminal in the system scenario, and each agent selects an action from the action space by considering the network status.
8. The resource allocation method for digital twin-assisted end-to-end deterministic latency network slices in the industrial Internet of Things according to claim 1 is characterized in that: In S5, the DT-PER-MASAC algorithm comprises the following steps: S81: Initialize the actor network parameters of the agent, the evaluation network parameters and target network parameters in the critic network, and the experience replay pool; S82: Determine whether the number of iterations exceeds the set number. If yes, stop the iteration. Otherwise, continue to execute S83. S83: For N samples in the experience replay pool, obtain the initial local state information of each agent from the DT-assisted E2E network slicing environment; S84: Determine whether the set number of training times is exceeded, if yes, stop the training, otherwise continue to execute S85; S85: Update the actor network and critic network according to the priority experience playback mechanism; S86: recalculate the time differential error TD-error value of the extracted experience, update the priority and temperature coefficient of the experience, and soft-update the target network parameters.
9. The resource allocation method for digital twin-assisted end-to-end deterministic latency network slices in the industrial Internet of Things according to claim 7 is characterized in that: The priority experience replay PER in the DT-PER-MASAC algorithm uses priority to calculate the probability of random selection of the sample, and the priority of sample i is expressed as: Among them, s t and t+1 are the states of time slot t and the next time slot, a t and a t+1 are the actions selected in time slot t and the next time slot, r(s t ,a t ) is based on the state s t Execute action a t Rewards received, and are the action value functions of the current time slot and the next time slot, α is the temperature coefficient, and π φ (a t+1 |s t+1 ) indicates that the strategy is in state s t+1 Take action a t+1 The probability of is a positive constant that approaches 0 infinitely, ensuring that every transition can be sampled even if the TD error is zero.
Citation Information
Patent Citations
Internet of vehicles network slice switching and resource allocation method based on digital twinning
CN117880892A
Digital twinning assisted edge slice resource allocation method in industrial internet of things
CN117915407A
Network resource deployment method and apparatus, and electronic device and storage medium
WO2023098374A1
Joint optimization system and method for computation offloading and resource allocation in multi-constraint-edge environment
WO2024065903A1
Cited By
Differentiated time slot access method for risk-sensitive state update service
CN119653495A
End-to-end deterministic slice arrangement method and device based on deep reinforcement learning
CN119789113A
End-to-end deterministic slice orchestration method and device based on deep reinforcement learning
CN119789113B
