Digital twinning auxiliary virtual network function active migration method based on resource prediction
Through real-time monitoring and resource prediction of digital twin technology, an active migration strategy for virtual network functions is formulated, which solves the service quality problems caused by untimely migration of virtual network functions and improves the performance and reliability of network systems.
Patent Information
- Application Number
- CN202410988740.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-23
AI Technical Summary
The untimely migration of virtual network functions in network slices leads to the inability to guarantee the service quality of network services, resulting in performance degradation and service interruption.
Through digital twin technology, real-time monitoring and collecting operation information of virtual network functions (VNF), using graph neural networks to predict resource demand, and formulate active migration strategies based on prediction results to optimize the deployment of virtual network functions to minimize the average delay and Resource variance.
Improve the ability to ensure service quality, avoid performance degradation and service interruption caused by NFV node overload, realize the optimization of virtual network function migration strategy when network resources are limited, and reduce the average delay and average resource variance of VNF. Improved the service performance of the network system.
Smart Images

Figure CN119030877A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of mobile communications and relates to a method for active migration of virtual network functions assisted by digital twins based on resource prediction. Background Art
[0002] As an emerging network architecture, network slicing has been widely regarded as a promising technology. Network slicing deploys a series of virtual network functions (VNFs) to form a service function chain (SFC) through network function virtualization (NFV) and software defined networking (SDN) to support flexible and efficient network resource allocation, thereby providing high-quality services for different businesses.
[0003] However, while network slicing brings flexibility, it also brings huge challenges to network resource management. Due to the dynamic changes in network traffic, untimely resource scheduling will make it difficult to ensure the continuity of network services. VNF migration is an effective technology to ensure the flexibility of network slicing and service continuity. It migrates VNFs from nodes with poor performance to nodes with better performance to avoid problems such as performance degradation and service interruption due to overload.
[0004] However, triggering passive VNF migration based on real-time network information will generate a certain migration delay, which will lead to service delay. In order to avoid the "lag" of service caused by passive migration, the VNF active migration strategy can be formulated in advance based on the predicted future resource requirements of the VNF, which can effectively solve the problem of migration strategy lag. Therefore, in order to properly utilize network slicing, it is crucial to effectively monitor the network and generate migration strategies based on accurate resource prediction results. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a digital twin-assisted virtual network function active migration method based on resource prediction, which monitors and collects VNF operation information in real time through digital twin technology, completes VNF resource prediction according to the digital twin model, and formulates VNF active migration strategy according to the resource prediction results, so as to avoid problems such as performance degradation and service interruption caused by NFV node overload, thereby improving the ability to ensure business service quality (QoS).
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] A digital twin-assisted virtual network function active migration method based on resource prediction, the method comprising:
[0008] S1. Build a digital twin-assisted virtual network function migration architecture, which includes the physical network layer, network slicing layer, and digital twin layer.
[0009] The physical network layer includes VNF nodes and digital twin service nodes, wherein the VNF nodes are used to process VNF services, and the digital twin service nodes are used to complete real-time monitoring and data collection of VNFs;
[0010] The network slicing layer deploys a series of SFCs on the VNF nodes through SDN / NFV technology to process different network service requests;
[0011] The digital twin layer constructs a global network topology through the digital twin service node to achieve real-time monitoring and management of VNFs, while providing auxiliary verification for VNF resource demand prediction and migration strategy formulation;
[0012] S2. Based on the collected information of the digital twin service nodes, a digital twin global network topology is constructed in the cloud, and the spatiotemporal information of the network topology is extracted using the graph neural network to predict VNF resource demand. Based on the prediction results, VNFs are actively migrated for nodes that are about to be overloaded or underloaded, so that the physical nodes maintain high performance and ensure the service QoS on the nodes.
[0013] S3. According to the VNF resource demand prediction results, a digital twin-assisted VNF active migration strategy based on multi-agent actor critic is formulated, and the optimization problem is established with the minimization of the VNF average latency and the average resource variance of the VNF nodes as the optimization goal;
[0014] S4. Convert the optimization problem into a Markov decision process, define the state space, action space and reward function, and use a multi-agent-based proximal optimization strategy to solve the optimal VNF digital twin deployment strategy.
[0015] Furthermore, in step S2, VNF resource requirements are predicted by designing a prediction model based on a graph convolutional neural network and a temporal convolutional network.
[0016] The prediction model includes a graph learning module, a GCN network and a TCN network; the graph learning module is used to adaptively generate a graph adjacency matrix; the TCN network extracts the temporal features of the VNF multi-resource data sequence through dilated time convolution; the temporal features output by the TCN network and the graph adjacency matrix reconstructed by the graph learning module are input into the GCN network to extract spatial features. When the SFC topology changes, the process is repeated again to achieve dynamic VNF resource prediction.
[0017] Furthermore, the graph learning module uses the hidden relationships between VNFs to adaptively generate a graph adjacency matrix: A=G gl (E) = Θ s (δ(M 1 M 2 T -M 2 M1 T ));where E represents a learnable node embedding with random initialization; M 1 =tanh(ωE 1 σ 1 ), M 2 =tanh(ωE 2 σ 2 ), where tanh(·) represents the tangent hyperbolic activation function, E 1 and E 2 represents node embedding, σ 1 , σ 2 represents the model parameters; δ(·) represents the elements used to regularize the adjacent matrix; Θ s represents the row-wise sparse function; G gl (·) represents the graph learning function.
[0018] Furthermore, in step S3, the optimization problem is expressed as:
[0019]
[0020]
[0021] In the formula, ξ 1 and 2 represents the weight factor, ξ 1 +ξ 2 =1,τ i Indicates the total delay threshold; and They represent the average latency and average resource load variance of VNFs in multiple time slots respectively; A binary variable indicating whether the VNF is mapped to a physical node, A binary variable indicating whether a virtual link is mapped to a physical link, A binary variable indicating whether the VNF is associated with a DT node, x, y, and Φ are A simplified representation of; F represents the set of service function chains, represents the set of VNFs on SFC i, represents the set of virtual links on SFC i, represents the virtual link between the jth and kth VNFs; L P represents the set of all links between nodes, N P Represents a set of physical nodes, l nm Indicates a physical link; and They represent the computing resources and storage resources required by VNF respectively; represents the bandwidth resources required by the virtual link in time slot t; C n、M n , B n,m They represent the computing resources, storage resources and bandwidth resources on the corresponding physical links of the physical nodes. They represent the upper limits of the thresholds of computing resources, storage resources, and bandwidth resources respectively;
[0022] Among the above constraints, constraint C1 ensures that each VNF can only be mapped to one physical node; constraint C2 ensures that each virtual link can only be mapped to one physical link; constraints C3 to C5 represent the computing, storage and bandwidth resource constraints of the physical node and the physical link respectively; constraint C6 represents the SFC end-to-end delay constraint; constraint C7 represents the binary variable constraint.
[0023] Further, in step S4, the established optimization problem is converted into a Markov decision process, and the state space, action space and reward function are defined;
[0024] For the state space S, let s(t) = {ψ(t), ξ(t), C(t), M(t), B(t)}∈S represent the network system state at time t, ψ(t) represents the network state information of the VNF node, ξ(t) represents the network state information of the link, C(t), M(t), B(t) represent the bandwidth resource demand state space of the VNF computing, storage and virtual link respectively;
[0025] For the action space A, define a(t)={a 1,1 (t),a 1,2 (t),...,a i,j (t),...}∈A represents the set of mapping actions that the VNF can take at time t;
[0026] For the reward function r(t), it is defined that the network system takes action a(t) in state s(t) to generate a mapping strategy π and obtain an instantaneous reward
[0027] According to the converted Markov process, the multi-agent-based proximal optimization strategy is used to solve the optimal VNF digital twin deployment strategy. The solution process includes:
[0028] 1) According to the VNF resource demand prediction results, calculate the resource utilization rate η of each VNF node R , determine whether the node is in an overloaded or underloaded state, and balance the node load by actively migrating VNFs;
[0029] 2) Initialize the Actor network parameters θ of each agent i , Critic network parameters and Experience Replay Pool
[0030] 3) The agent obtains network status information s i,t , and use the Actor network according to s i,t Get Action Indicates that the policy network is in state s i,t Next select action a i,t The probability distribution of i are the parameters of the policy network;
[0031] 4) The agent gets the reward r for the corresponding VNF mapping and resource allocation according to the action i,t , and obtain the next local state information s i,t+1 ;
[0032] 5) Get global status information s t , global action information a t and the next step global status information s t+1 ;
[0033] 6)Tuple(s t ,a t ,r i,t ,s t+1 ) stored in the experience replay pool from Randomly take small batches of experience data {(s j ,a j ,r i,j ,s j+1 )} as training data, and use gradient descent to update the Actor network, Critic network and temperature coefficient, and use soft update to update the target value network;
[0034] 7) The loop is executed until the reward function converges or the maximum number of training rounds is reached.
[0035] The beneficial effects of the present invention are as follows: the present invention aims at the problem that the quality of network service cannot be guaranteed due to untimely migration of virtual network functions. The present invention uses digital twin technology to monitor and collect VNF operation information in real time, completes VNF resource prediction based on the digital twin model, and formulates VNF active migration strategy based on the resource prediction results, so as to avoid problems such as performance degradation and service interruption caused by NFV node overload, thereby improving the ability to ensure service QoS. The present invention can optimize the virtual network function migration strategy, reduce the average delay and average resource variance of VNF, and improve the service performance of the network system when network resources are limited.
[0036] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0038] Figure 1 This is a schematic diagram of the architecture of the digital twin-assisted virtual network function migration proposed in the present invention;
[0039] Figure 2 It is the DT-TGN resource prediction model;
[0040] Figure 3 This is a framework diagram of the virtual network function migration algorithm based on DT-MASAC proposed in the present invention. DETAILED DESCRIPTION
[0041] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0042] An embodiment of the present invention provides a digital twin-assisted virtual network function active migration method based on resource prediction. The method uses digital twin technology to monitor and collect VNF operation information in real time, completes VNF resource prediction based on the digital twin model, and formulates a VNF active migration strategy based on the resource prediction results, so as to avoid problems such as performance degradation and service interruption caused by NFV node overload, thereby improving the ability to ensure business QoS.
[0043] Specifically, the method is implemented as follows:
[0044] S1: Build a digital twin-assisted virtual network function migration architecture, such as Figure 1 As shown in the figure, the architecture includes: physical network layer, network slicing layer and digital twin layer.
[0045] The physical network layer selects a part of physical nodes for deploying and processing VNF services, and another part of physical nodes is used as digital twin service nodes for building VNF digital twins and real-time monitoring and collection of VNF operation information.
[0046] The network slicing layer mainly uses NFV technology to flexibly deploy a series of VNFs at the physical network layer to handle different network service requests.
[0047] The digital twin layer constructs the global network topology through the data collected by the digital twin server, uses the graph neural network to fully extract the spatiotemporal information of the network topology, completes the resource prediction of the VNF, and formulates the VNF active migration strategy based on the resource prediction results.
[0048] The physical layer is defined as a fully connected undirected graph G P =(N P ,L P ), N P Represents a collection of physical nodes, including VNF nodes and digital twin service nodes And the same node cannot be deployed with both VNF nodes and DT nodes. P Represents the set of all links between nodes. Assume that each node has certain computing resources and storage resources. The computing resources and storage resources of the VNF node are represented as C n and M n The computing resources and storage resources of the digital twin service node are represented as and Assume l nm ∈L P Represents the link between nodes, and the bandwidth capacity on the link is represented by B n,m The network slice layer deploys SFCs, which are composed of a series of VNFs in sequence. SFCs are responsible for different business processing. Definition represents the i-th SFC on the network slice, which is expressed as The set of SFC is F={f 1 ,f 2 ,…,f i ,…}, is the set of VNFs on SFC i, is the set of virtual links on SFC i. and They are defined as the computing resource requirements and storage resource requirements of each VNF, Virtual Link In addition, this paper assumes A binary variable indicating whether VNF j on SFC i is mapped to physical node n. This means that VNF j is mapped to physical node n, otherwise Indicates a virtual link Whether to map to physical link l nm If Virtual Link Mapping to physical link l nm On, otherwise
[0049] Modeling the Digital Twin of VNF as DT f ={D n ,M n ,S n (t),ΔS n (t+1)}, where D n Represented as the historical data of VNF digital twin, M n represents the digital twin model of VNF on the physical node, S n (t) represents the operating state of VNFf at time slot t, ΔS n (t+1) represents the status update of VNFf. Building and maintaining VNF digital twins in digital twin service nodes also consumes physical resources. represents the CPU resources required to build the digital twin of VNF on the DT node. Represents the storage resources required to build and maintain the digital twin of VNFf on the DT node. Represents the bandwidth resources required to transmit the VNFf digital twin data through the link.
[0050] SFC affects network performance and services by end-to-end delay. For each SFC, the VNF processing delay, transmission delay on NFV nodes, and link propagation delay are mainly considered. The details are as follows:
[0051] At time slot t, the transmission delay of VNF between nodes is expressed as: Among them, d i (t) represents the packet size of the VNF on the transmission path at time slot t, Indicates whether the t-time slot virtual link is mapped to the physical link. Indicates the bandwidth resources required by the t-time slot virtual link.
[0052] The propagation delay of a link can be expressed as: Where D nm Is the physical link l nm distance, c(t) represents the speed of light.
[0053] Then the total link delay of VNF transmitted from NFV node m to physical node n in time slot t can be expressed as:
[0054] VNF processing latency is the processing delay of node n in processing VNF in time slot t, expressed as:
[0055]
[0056] in Indicates the processing rate of the node for VNF, Indicates the processing rate coefficient of the data packet. Indicates whether the VNF in time slot t is mapped to physical node n.
[0057] In summary, the total delay on SFC can be expressed as:
[0058]
[0059] Among them, F represents the set of service function chains, is the set of VNFs on SFC i, is the set of virtual links on SFC i, represents the virtual link between the jth and kth VNFs.
[0060] Then the average latency of all VNF instances running in time slot t can be expressed as:
[0061]
[0062] Among them, |N i | represents the number of VNFs on SFC i, so the average delay of VNFs in q time slots can be expressed as
[0063] Load balancing of physical nodes after VNF migration helps the network system to be more robust in the face of traffic changes. Load balancing of physical nodes is a long-term and effective use of network resources. VNF migrates from node n to node m at time slot t, which causes a load change on node m. This change is caused by the consumption of computing and storage resources of node m. At this time, the load of node m can be expressed as:
[0064]
[0065] in, and They represent the computing resource load and storage resource load of physical node m respectively, Indicates whether the VNF is mapped to the physical node m. They represent the computing and storage resources required by VNF respectively. The mean values of computing resource load and storage resource load of physical node m are respectively expressed as:
[0066]
[0067] Among them, C m and M m Represents the computing and storage resources owned by physical node m. The load balancing capability of the network system can be measured by the average network resource variance of the physical node. The average computing variance and average storage variance of the physical resources can be expressed as:
[0068]
[0069]
[0070] Load capacity of the network system L total (t) can be obtained by the weighted sum of the variance of computing and storage resources, which is expressed as follows:
[0071]
[0072] Among them, ω 1 and ω 2 They represent the weight factors of computing resources and storage resources on the load balancing of the network system. Here, ω 1 +ω 2 =1. Then, the average total resource load variance in q time slots can be expressed as
[0073] S2: Under the multi-dimensional resource constraints of computing, memory, and bandwidth, the optimization problem is established with the goal of minimizing the average latency and average load resource variance of VNFs. The details are as follows:
[0074]
[0075] In the formula, ξ 1 and 2 is the weight factor, τ i is the total delay threshold, x, y, Φ are A simplified representation of , where A binary variable indicating whether the VNF is mapped to a physical node, A binary variable indicating whether a virtual link is mapped to a physical link, A binary variable indicating whether the VNF is associated with a DT node; C n 、M n , B n,m They represent the computing resources, storage resources and bandwidth resources on the corresponding physical links of the physical nodes. They are the upper limits of the thresholds for computing resources, storage resources, and bandwidth resources.
[0076] The optimization goal is subject to the constraints of C1 to C7 to ensure the effectiveness of the optimization goal: C1 ensures that each VNF can only be mapped to one physical node; C2 ensures that each virtual link can only be mapped to one physical link; C3-C4 represent the computing, storage and bandwidth resource constraints of the physical node and the physical link respectively; C6 represents the SFC end-to-end delay constraint; C7 is a series of binary variable constraints.
[0077] Further, in step S2, VNF resource demand prediction is performed based on the resource usage data collected by the digital twin and the global SFC topology, as follows:
[0078] Design a digital twin-assisted prediction model (DT-TCN) based on graph convolutional neural network (GCN) and temporal convolutional network (TCN), such as Figure 2 As shown in the figure, the model fully extracts spatiotemporal features based on the global SFC topology of the digital twin to improve the accuracy of VNF resource demand prediction. The prediction model includes a graph learning module, a GCN network and a TCN network.
[0079] The digital twin collects a large amount of data from the physical network layer in real time, including computing and storage on VNF nodes, bandwidth on links and other multi-dimensional resource information. is the set of VNF nodes, and E is the set of virtual links. The adjacency matrix of in The resource data of each VNF on the SFC collected by DT can be expressed as a multivariate time series, expressed as Where D represents the dimension of the feature. The multivariate time series with a graph structure can be considered as a dynamic graph. At each time t, the resource usage of each VNF can be expressed as x t ={x 1 ,x 2 ,...,x n Resource prediction aims to predict the state of the graph at the next moment in the future based on this historical information. This can be achieved by learning a function f(·) that maps the graph state at a given previous T time steps to the graph state at the next time step. It is expressed as:
[0080] Graph learning module: Capturing spatial correlation based on graph neural networks mostly relies on predefined graph structures. However, the deployment of VNFs and the graph structure of SFCs will change dynamically according to the network environment. Therefore, it is necessary to mine the dependencies between VNF nodes from the data and update the graph structure during the training process. The present invention proposes a graph learning module to adaptively generate a graph adjacency matrix and make full use of the hidden relationships between VNFs, which is expressed as:
[0081] A=G gl (E) = Θ s (δ(M 1 M 2 T -M 2 M 1 T ))
[0082] in represents a learnable node embedding with random initialization, d is the node embedding dimension, and each element A i,j is the embedding E from the corresponding node i and E j Recovered in It is converted by multi-layer perceptron, where M 1 =tanh(ωE 1 σ 1 )、M 2 =tanh(ωE 2 σ 2 ), tanh(·) represents the tangent hyperbolic activation function, E 1 and E 2 represents node embedding, σ 1 , σ 2 are model parameters, δ(·) = ReLU (tanh(ω(·)) is used to regularize the elements of the adjacent matrix, ω controls the saturation rate of the activation function, Θ s is a row-wise sparse function. In addition, M 1 M 2 T -M 2 M 1 T To achieve the asymmetry of matrix A, so as to construct an adjacent matrix that is more in line with the dynamic changes of SFC topology in the actual network. gl (·) represents the graph learning function, through which the adjacency matrix is obtained.
[0083] Time feature extraction module: In order to extract the long-term changing VNF operation data with short sampling interval and large data volume, TCN design is used to extract the global information of time series. Given a time series X = (x 1 ,x 2 ,...,xt ) and convolution filter F = (f 1×1 ,f 1×2 ,...,f 1×l ), x t The dilated convolution at can be expressed as: in represents the convolution operation, d is the dilation factor that increases with the number of network layers, l is the filter kernel size, and the subscript x t-i·d Indicates the direction of the past.
[0084] By expanding the receptive field to extract long-term information, the dilation factor of each temporal convolutional layer is set to grow exponentially at a rate of b (b>1). Given the receptive field w=1+d(l-1), and the initial dilation factor is set to 1, the size of the dilated receptive field is expressed as: where b d Represents the value of the dilation factor at the dth layer.
[0085] Spatial feature extraction module: After the graph learning module updates the adjacency matrix, the temporal features output by the temporal feature extraction module and the graph adjacency matrix reconstructed by graph learning are used as input, and the spatial propagation module based on GCN is used to extract spatial features. Let k represent the index of each spatial time layer. After reaching this module through each layer of the extended convolution module, the aggregation of neighbor node information is completed based on the data, and the representation of the node data is updated: In the kth layer of the convolution module, the input after graph learning and temporal convolution is represented as H = Z (k) , then the layered propagation rule of graph convolution is expressed as: in represents the adjacency matrix of an undirected graph G with additional self-connections, I N is the identity matrix, is the trainable weight matrix for a particular layer. is a diagonal matrix with self-loops, and are the node representations in the convolutional layers of the kth and k+1th graphs, respectively, and W (k) is the parameter matrix and σ is the nonlinear activation function.
[0086] In the final output layer, the skip connection layer is actually a 1×L k A standard convolutional layer, where L k represents the input length of the kth jump connection layer, and normalizes the information jumping to the output module so that they have the same output sequence length. The output module consists of two one-dimensional standard convolution modules, which convert the final prediction result Y i ={y 1 ,y 2 ,...,y n} to the desired output dimension sequence.
[0087] S3. For the proposed optimization problem P, the optimization problem is modeled as a Markov process, the state space, action and reward functions are defined, and the multi-agent-based proximal optimization strategy is used to solve the optimal VNF digital twin deployment strategy. Figure 3 The figure shows the framework diagram of the VNF active migration algorithm based on DT-MASAC. The details are as follows:
[0088] For the state space S, define s(t) = {ψ(t), ξ(t), C(t), M(t), B(t)}∈S to represent the network system state at time t, where ψ(t) is the network state information of the VNF node, ξ(t) is the network state information of the link, C(t), M(t), and B(t) represent the bandwidth resource demand state space of the VNF's computing, storage, and virtual links, respectively.
[0089] For the action space A, define a(t) = {a 1,1 (t),a 1,2 (t),...,a i,j (t),...}∈A represents the set of mapping actions that the VNF can take at time t, where a i,j (t) represents the mapping action of the j-th VNF on SFC i.
[0090] The transition probability p(s(t+1)|s(t),a(t)) represents the probability of transitioning to the next state s(t+1) after taking action a(t)∈A in the current state s(t)∈S.
[0091] When the network system takes action a(t) in state s(t), it will generate a mapping strategy π, which can obtain an instantaneous reward If the constraint is not met, the reward r(t) = -1 / ε, where ε is the penalty factor.
[0092] Further, in step S3, the DT-MASAC migration algorithm is specifically as follows:
[0093] The proposed DT-MASAC algorithm is an extension of the SAC algorithm in the multi-agent field using a centralized training distributed execution framework. Taking each VNF as an agent, each agent interacts with the VNF digital twin and obtains the state information of the corresponding nodes and links from the digital twin environment, as well as the bandwidth resource requirements of the VNF's computing, storage, and virtual links. The agent then takes the corresponding VNF mapping action and calculates the instantaneous reward. In the centralized training stage, the input of the value network of each agent includes not only its own local state information, but also the local state information of the agent. i (t) and action information a i (t), and also includes the state information of other agents o-i (t) and action information a -i (t). The policy network only needs to input the local state information of the agent to output the policy distribution, thereby obtaining actions through sampling. In the execution phase, each agent only needs to use the policy network to execute independently. Define the global state information as s t ={o i (t),o -i (t)}, and define the global action information as a t ={a i (t),a -i (t)}. By introducing the maximum entropy method, the exploration and stability of the algorithm are improved. The goal is to maximize the cumulative reward and policy entropy. The objective function is expressed as:
[0094]
[0095] where H(π(·|s t ))=-logπ(·|s t ) represents the policy entropy, π(·|s t ) represents the probability distribution of the mapping from state to action. α∈(0,1) represents the temperature coefficient, which determines the relative importance of the policy entropy term to the reward, thereby controlling the randomness of the optimal policy. π represents the state-action distribution under policy π.
[0096] The centralized Q value can be defined as:
[0097]
[0098] where γ∈(0,1) is the discount factor, V(s t ) is expressed as:
[0099] In the algorithm, each agent uses two value networks and The bias caused by overestimation of Q value can be reduced. The two value networks have the same network structure and do not share parameters. The value network is trained to minimize the soft Bellman residual, which can be expressed as:
[0100]
[0101] in represents the experience replay buffer, φ i,n is the parameter of the value network, which can be obtained by minimizing the loss function J Q (φ i,n ) to update, are the parameters of the target value network. The value network is trained by sampling experiences from the experience replay pool, and the target network can be updated periodically as Indicates that the policy network is in state s i,t+1 Next select action a i,t+1 The probability distribution of i are the parameters of the policy network.
[0102] The parameters of the policy network are i It can be updated by minimization. In order to facilitate the calculation of gradients, the reparameterization technique can be used, and the reparameterization technique can produce a lower variance estimate. i,t Reparameterize to where ∈ i,t is an input noise vector that can be sampled from some fixed distribution. So J π (θ i ) is expressed as:
[0103]
[0104] In the formula represents the noise distribution, which is a Gaussian distribution here. In addition, a target strategy entropy is set The temperature coefficient α is updated by minimizing the entropy loss function J(α):
[0105]
[0106] In summary, the execution steps of the DT-MASAC algorithm are as follows:
[0107] 1) According to the VNF resource demand prediction results, calculate the resource utilization rate η of each VNF node R , determine whether the node is in an overloaded or underloaded state, and balance the node load by actively migrating VNFs;
[0108] 2) Initialize the Actor network parameters θ of each agent i , Critic network parameters and Experience Replay Pool
[0109] 3) The agent obtains network status information s i,t , and use the Actor network according to s i,t Get Action
[0110] 4) The agent gets the reward r for the corresponding VNF mapping and resource allocation according to the action i,t , and obtain the next local state information s i,t+1 ;
[0111] 5) Get global status information s t , global action information a t and the next step global status information s t+1 ;
[0112] 6)Tuple(s t ,a t ,r i,t ,s t+1 ) stored in the experience replay pool from Randomly take small batches of experience data {(s j ,a j ,r i,j ,s j+1 )} as training data, and use gradient descent to update the Actor network, Critic network and temperature coefficient, and use soft update to update the target value network. The loop is executed until the reward function converges or the number of training rounds reaches the maximum value.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A digital twin-assisted active migration method for virtual network functions based on resource prediction, Features: The method comprises the following steps: S1. Build a digital twin-assisted virtual network function migration architecture, which includes a physical network layer, a network slicing layer and a digital twin layer; the physical network layer includes VNF nodes and digital twin service nodes, the VNF nodes are used to process VNF services, and the digital twin service nodes are used to complete real-time monitoring and data collection of VNFs; the network slicing layer deploys a series of SFCs on the VNF nodes through SDN / NFV technology to process different network service requests; the digital twin layer builds a global network topology through the digital twin service nodes to achieve real-time monitoring and management of VNFs, and at the same time provides auxiliary verification for VNF resource demand prediction and migration strategy formulation; S2. Based on the collected information of the digital twin service node, a digital twin global network topology is constructed in the cloud, and the spatiotemporal information of the network topology is extracted using a graph neural network to predict VNF resource demand; based on the prediction results, VNFs are actively migrated for nodes that are about to be overloaded or underloaded, so that the physical nodes maintain high performance and ensure the service QoS on the nodes; S3. According to the VNF resource demand prediction results, a digital twin-assisted VNF active migration strategy based on multi-agent actor critic is formulated, and the optimization problem is established with the minimization of the VNF average latency and the average resource variance of the VNF nodes as the optimization goal; S4. Convert the optimization problem into a Markov decision process, define the state space, action space and reward function, and use a multi-agent-based proximal optimization strategy to solve the optimal VNF digital twin deployment strategy.
2. The method according to claim 1, Features: In step S2, VNF resource requirements are predicted by designing a prediction model based on graph convolutional neural network and temporal convolutional network.
3. The method according to claim 2, Features: The prediction model includes a graph learning module, a GCN network and a TCN network; the graph learning module is used to adaptively generate a graph adjacency matrix; the TCN network extracts the time characteristics of the VNF multi-resource data sequence by dilating the time convolution; The temporal features output by the TCN network and the graph adjacency matrix reconstructed by the graph learning module are input into the GCN network to extract spatial features.
4. The method according to claim 3, Features: The graph learning module uses the hidden relationships between VNFs to adaptively generate a graph adjacency matrix: A = G gl (E) = Θ s (δ(M 1 M 2 T -M 2 M 1 T ));where E represents a learnable node embedding with random initialization; M 1 =tanh(ωE 1 σ 1 ), M 2 =tanh(ωE 2 σ 2 ), where tanh(·) represents the tangent hyperbolic activation function, E 1 and E 2 represents node embedding, σ 1 , σ 2 represents the model parameters; δ(·) represents the elements used to regularize the adjacent matrix; Θ s represents the row-wise sparse function; G gl (·) represents the graph learning function.
5. The method according to claim 1, Features: In step S3, the optimization problem is expressed as: In the formula, ξ 1 and 2 represents the weight factor, ξ 1 +ξ 2 =1,τ i Indicates the total delay threshold; and They represent the average latency and average resource load variance of VNFs in multiple time slots respectively; A binary variable indicating whether the VNF is mapped to a physical node, A binary variable indicating whether a virtual link is mapped to a physical link, A binary variable indicating whether the VNF is associated with a DT node, x, y, and Φ are A simplified representation of; F represents the set of service function chains, represents the set of VNFs on SFC i, represents the set of virtual links on SFC i, represents the virtual link between the jth and kth VNFs; L P represents the set of all links between nodes, N P Represents a set of physical nodes, l nm Indicates a physical link; and They represent the computing resources and storage resources required by VNF respectively; represents the bandwidth resources required by the virtual link in time slot t; C n 、M n , B n,m They represent the computing resources, storage resources owned by the physical node, and the bandwidth resources on the corresponding physical link. They represent the upper threshold limits of computing resources, storage resources, and bandwidth resources respectively; Constraint C1 ensures that each VNF can only be mapped to one physical node; constraint C2 ensures that each virtual link can only be mapped to one physical link; constraints C3 to C5 represent the computing, storage and bandwidth resource constraints of the physical node and the physical link respectively; constraint C6 represents the SFC end-to-end delay constraint; constraint C7 represents the binary variable constraint.
6. The method according to claim 5, Features: Convert the established optimization problem into a Markov decision process, define the state space, action space, and reward function; For the state space S, let s(t) = {ψ(t), ξ(t), C(t), M(t), B(t)}∈S represent the network system state at time t, ψ(t) represents the network state information of the VNF node, ξ(t) represents the network state information of the link, C(t), M(t), B(t) represent the bandwidth resource demand state space of the VNF computing, storage and virtual link respectively; For the action space A, define a(t)={a 1,1 (t),a 1,2 (t),...,a i,j (t),...}∈A represents the set of mapping actions that the VNF can take at time t, where a i,j (t) represents the mapping action of the jth VNF on SFC i; For the reward function r(t), it is defined that the network system takes action a(t) in state s(t) to generate a mapping strategy π and obtain an instantaneous reward 7. The method according to claim 6, Features: According to the converted Markov process, the multi-agent-based proximal optimization strategy is used to solve the optimal VNF digital twin deployment strategy. The solution process includes: 1) According to the VNF resource demand prediction results, calculate the resource utilization rate η of each VNF node R , determine whether the node is in an overloaded or underloaded state, and balance the node load by actively migrating VNFs; 2) Initialize the Actor network parameters θ of each agent i , Critic network parameters and Experience Replay Pool 3) The agent obtains network status information s i,t , and use the Actor network according to s i,t Get Action Indicates that the policy network is in state s i,t Next select action a i,t The probability distribution of i are the parameters of the policy network; 4) The agent gets the reward r for the corresponding VNF mapping and resource allocation according to the action i,t , and obtain the next local state information s i,t+1 ; 5) Get global status information s t , global action information a t and the next step global status information s t+1 ; 6)Tuple(s t ,a t ,r i,t ,s t+1 ) stored in the experience replay pool from Randomly take small batches of experience data {(s j ,a j ,r i,j ,s j+1 )} as training data, and use gradient descent to update the Actor network, Critic network and temperature coefficient, and use soft update to update the target value network; 7) The loop is executed until the reward function converges or the maximum number of training rounds is reached.
Citation Information
Patent Citations
Virtual network function migration method for digital twinning assisted dynamic resource demand prediction
CN116137593A
Virtual network function migration method in digital twinning assisted network slice
CN117979366A
Cited By
VNF multi-dimensional resource demand prediction method based on synchronous and asynchronous space-time characteristic fusion
CN119740095A
A VNF-based multidimensional resource demand forecasting method based on the fusion of synchronous and asynchronous spatiotemporal characteristics.
CN119740095B
Low-delay slice resource intelligent allocation method for 5G network
CN120302446A
Data migration task scheduling method and device, equipment, storage medium and product
CN120475033A
