A resource prediction-based digital twin assisted virtual network function active migration method

Abstract

The present application relates to a kind of digital twin assisted virtual network function active migration method based on resource prediction, belong to mobile communication technical field.The method includes: constructing digital twin assisted virtual network function migration architecture, and real-time monitoring and collecting VNF operation information by digital twin technology.According to the acquisition information of digital twin, construct digital twin global network topology, utilize graph neural network to fully extract the space-time information of network topology, carry out VNF resource demand prediction, according to the prediction result, complete VNF active migration to the node to be overloaded or lightly loaded.According to the prediction result of VNF resource demand, formulate digital twin assisted VNF active migration strategy based on multi-agent actor critic.The present application can realize the optimization of virtual network function migration strategy under the condition of limited network resources, reduce VNF average time delay and average resource variance, improve the service performance of network system.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile communications and relates to a digital twin-assisted virtual network function active migration method based on resource prediction. Background Art

[0002] Network slicing, an emerging network architecture, is widely considered a promising technology. Through network function virtualization (NFV) and software-defined networking (SDN), network slicing deploys a series of virtual network functions (VNFs) to form a service function chain (SFC). This supports flexible and efficient network resource allocation, thereby providing high-quality services for different businesses.

[0003] However, while network slicing brings flexibility, it also presents significant challenges for network resource management. Due to the dynamic nature of network traffic, untimely resource scheduling can make it difficult to ensure network service continuity. VNF migration is an effective technology for ensuring network slicing flexibility and service continuity. It migrates VNFs from low-performing nodes to high-performing ones, thus avoiding performance degradation and service interruptions caused by overload.

[0004] However, triggering passive VNF migration based on real-time network information can result in a certain migration delay, leading to service delays. To avoid service lags caused by passive migration, proactive VNF migration strategies can be developed based on forecasts of future VNF resource requirements. This effectively addresses this migration delay issue. Therefore, effective network monitoring and migration strategies based on accurate resource forecasts are crucial for the effective utilization of network slicing. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a digital twin-assisted virtual network function active migration method based on resource prediction, which monitors and collects VNF operation information in real time through digital twin technology, completes VNF resource prediction based on the digital twin model, and formulates VNF active migration strategy based on the resource prediction results, so as to avoid problems such as performance degradation and service interruption caused by NFV node overload, thereby improving the ability to ensure business service quality (QoS).

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A digital twin-assisted virtual network function active migration method based on resource prediction, the method comprising:

[0008] S1. Build a digital twin-assisted virtual network function migration architecture, which includes the physical network layer, network slicing layer, and digital twin layer.

[0009] The physical network layer includes VNF nodes and digital twin service nodes, where VNF nodes are used to process VNF services and digital twin service nodes are used to complete real-time monitoring and data collection of VNFs;

[0010] The network slicing layer deploys a series of SFCs on the VNF nodes through SDN / NFV technology to handle different network service requests;

[0011] The digital twin layer constructs a global network topology through the digital twin service nodes to achieve real-time monitoring and management of VNFs, while providing auxiliary verification for VNF resource demand prediction and migration strategy formulation;

[0012] S2: Based on the information collected from the digital twin service nodes, a digital twin global network topology is constructed in the cloud. Graph neural networks are used to extract the spatiotemporal information of the network topology and predict VNF ​​resource demand. Based on the prediction results, VNFs are proactively migrated to nodes that are about to be overloaded or underloaded, ensuring high performance of physical nodes and guaranteed QoS for services on the nodes.

[0013] S3. Based on the VNF resource demand prediction results, a digital twin-assisted multi-agent actor-critic-based VNF active migration strategy is formulated, with the optimization problem established with the minimization of the VNF average latency and the average resource variance of the VNF nodes as the optimization objectives.

[0014] S4. Convert the optimization problem into a Markov decision process, define the state space, action space, and reward function, and use a multi-agent-based proximal optimization strategy to solve the optimal VNF digital twin deployment strategy.

[0015] Furthermore, in step S2, VNF resource requirements are predicted by designing a prediction model based on graph convolutional neural network and temporal convolutional network.

[0016] The prediction model consists of a graph learning module, a GCN network, and a TCN network. The graph learning module adaptively generates a graph adjacency matrix. The TCN network extracts temporal features of VNF multi-resource data sequences through dilated temporal convolution. The temporal features output by the TCN network and the graph adjacency matrix reconstructed by the graph learning module are input into the GCN network to extract spatial features. This process repeats itself when the SFC topology changes, enabling dynamic VNF resource prediction.

[0017] Furthermore, the graph learning module uses the hidden relationships between VNFs to adaptively generate a graph adjacency matrix: A=G gl (E) = Θ s (δ(M1M2 T -M2M1 T)); where E represents a learnable node embedding with random initialization; M1 = tanh(ωE1σ1), M2 = tanh(ωE2σ2), where tanh(·) represents the tangent hyperbolic activation function, E1 and E2 represent node embeddings, σ1 and σ2 represent model parameters; δ(·) represents the element used to regularize the adjacency matrix; Θ s represents the row-wise sparse function; G gl (·) represents the graph learning function.

[0018] Furthermore, in step S3, the optimization problem is expressed as:

[0019]

[0020]

[0021] Where ξ1 and ξ2 represent weight factors, ξ1+ξ2=1, τ i Indicates the total delay threshold; and Represents the average VNF latency and average resource load variance in multiple time slots; A binary variable indicating whether the VNF is mapped to a physical node. A binary variable indicating whether a virtual link is mapped to a physical link, Binary variables indicating whether the VNF is associated with the DT node, x, y, and Φ are Simplified representation of; F represents the set of service function chains, represents the set of VNFs on SFC i, represents the set of virtual links on SFC i, represents the virtual link between the jth and kth VNFs; L P represents the set of all links between nodes, N P Represents a set of physical nodes, l nm Indicates a physical link; and They represent the computing resources and storage resources required by VNF respectively; C represents the bandwidth resources required by the t-time slot virtual link; n 、M n 、B n,m They represent the computing resources, storage resources and bandwidth resources on the corresponding physical links owned by the physical nodes. Respectively represent the upper thresholds of computing resources, storage resources, and bandwidth resources;

[0022] Among the above constraints, constraint C1 ensures that each VNF can only be mapped to one physical node; constraint C2 ensures that each virtual link can only be mapped to one physical link; constraints C3 to C5 represent the computing, storage, and bandwidth resource constraints of the physical node, respectively; constraint C6 represents the SFC end-to-end delay limit; and constraint C7 represents a binary variable constraint.

[0023] Furthermore, in step S4, the established optimization problem is converted into a Markov decision process, and the state space, action space and reward function are defined;

[0024] For the state space S, let s(t) = {ψ(t), ξ(t), C(t), M(t), B(t)}∈S represent the network system state at time t, ψ(t) represents the network state information of the VNF node, ξ(t) represents the network state information of the link, C(t), M(t), and B(t) represent the state space of the VNF's computing, storage, and virtual link bandwidth resource requirements, respectively;

[0025] For the action space A, define a(t)={a 1,1 (t),a 1,2 (t),...,a i,j (t),...}∈A represents the set of mapping actions that the VNF can take at time t;

[0026] For the reward function r(t), it is defined that the network system takes action a(t) in state s(t) to generate a mapping strategy π and obtain an instantaneous reward

[0027] Based on the converted Markov process, a multi-agent-based proximal optimization strategy is used to solve the optimal VNF digital twin deployment strategy. The solution process includes:

[0028] 1) Calculate the resource utilization η of each VNF node based on the VNF resource demand prediction results R , determine whether the node is in an overloaded or underloaded state, and balance the node load by actively migrating VNFs;

[0029] 2) Initialize the Actor network parameters θ of each agent i , Critic network parameters and Experience Replay Pool

[0030] 3) The agent obtains network status information s i,t , and use the Actor network according to s i,t Get Action Indicates that the policy network is in state s i,t Next select action ai,t The probability distribution of θ i are the parameters of the policy network;

[0031] 4) The agent gets the reward r for the corresponding VNF mapping and resource allocation based on the action i,t , and obtain the next local state information s i,t+1 ;

[0032] 5) Get global status information s t , global action information a t and the next global status information s t+1 ;

[0033] 6)Tuple(s t ,a t ,r i,t ,s t+1 ) stored in the experience replay pool from Randomly take small batches of experience data {(s j ,a j ,r i,j ,s j+1 )} as training data, and use gradient descent to update the Actor network, Critic network and temperature coefficient, and update the target value network according to the soft update method;

[0034] 7) The loop is executed until the reward function converges or the maximum number of training rounds is reached.

[0035] The beneficial effects of the present invention are as follows: To address the problem of untimely virtual network function migration leading to unguaranteed network service quality, the present invention uses digital twin technology to monitor and collect VNF ​​operational information in real time, perform VNF resource prediction based on the digital twin model, and formulate a proactive VNF migration strategy based on the resource prediction results. This avoids problems such as performance degradation and service interruption caused by NFV node overload, thereby improving the ability to ensure service QoS. In the case of limited network resources, the present invention can optimize the virtual network function migration strategy, reduce the average VNF latency and average resource variance, and improve the service performance of the network system.

[0036] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0038] Figure 1 This is a schematic diagram of the architecture of the digital twin-assisted virtual network function migration proposed in the present invention;

[0039] Figure 2 It is the DT-TGN resource prediction model;

[0040] Figure 3 This is a framework diagram of the virtual network function migration algorithm based on DT-MASAC proposed in this invention. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0042] An embodiment of the present invention provides a digital twin-assisted virtual network function active migration method based on resource prediction. The method uses digital twin technology to monitor and collect VNF ​​operation information in real time, completes VNF resource prediction based on the digital twin model, and formulates a VNF active migration strategy based on the resource prediction results, so as to avoid problems such as performance degradation and service interruption caused by NFV node overload, thereby improving the ability to ensure service QoS.

[0043] Specifically, the method is implemented as follows:

[0044] S1: Build a digital twin-assisted virtual network function migration architecture, such as Figure 1 As shown in Figure 1, the architecture includes: physical network layer, network slicing layer and digital twin layer.

[0045] The physical network layer selects a part of physical nodes for deploying and processing VNF services, and another part of physical nodes as digital twin service nodes for building VNF digital twins and real-time monitoring and collection of VNF operation information.

[0046] The network slicing layer mainly uses NFV technology to flexibly deploy a series of VNFs at the physical network layer to handle different network service requests.

[0047] The digital twin layer constructs the global network topology through the data collected by the digital twin server, uses the graph neural network to fully extract the spatiotemporal information of the network topology, completes the resource prediction of the VNF, and formulates the VNF active migration strategy based on the resource prediction results.

[0048] The physical layer is defined as a fully connected undirected graph G P =(N P ,L P ), N P Represents a collection of physical nodes, including VNF nodes and digital twin service nodes And the same node cannot be deployed with VNF nodes and DT nodes at the same time. P Represents the set of all links between nodes. Assume that each node has a certain amount of computing resources and storage resources, and denote the computing resources and storage resources of the VNF node as C n and M n The computing resources and storage resources of the digital twin service node are represented as and Assume l nm ∈L P Represents the link between nodes, and the bandwidth capacity on the link is represented by B n,m The network slice layer deploys SFCs, which are composed of a series of VNFs in sequence. SFCs are responsible for different business processing. represents the i-th SFC on the network slice, which is expressed as The set of SFC is F={f1,f2,…,f i ,…}, is the set of VNFs on SFC i, is the set of virtual links on SFC i. and They are defined as the computing resource requirements and storage resource requirements of each VNF respectively. Virtual Link In addition, this paper assumes A binary variable indicating whether VNF j on SFC i is mapped to physical node n. It means that VNF ​​j is mapped to physical node n, otherwise Indicates a virtual link Is it mapped to physical link l? nm If the binary variable It indicates a virtual link Mapping to physical link l nm On, otherwise

[0049] Modeling the Digital Twin of VNF as DT f ={D n ,M n ,S n (t),ΔS n (t+1)}, where D n Represented as the historical data of VNF digital twin, M n Represents the digital twin model of VNF on the physical node, S n (t) represents the running state of VNFf at time slot t, ΔS n (t+1) represents the status update of VNFf. Building and maintaining VNF digital twins in digital twin service nodes also consumes physical resources. represents the CPU resources required to build the digital twin of the VNF on the DT node. Represents the storage resources required to build and maintain the digital twin of VNFf on the DT node. Indicates the bandwidth resources required to transmit VNFf digital twin data through the link.

[0050] SFC affects network performance and services by end-to-end latency. For each SFC, the main considerations are VNF processing latency, transmission latency on NFV nodes, and link propagation latency. The details are as follows:

[0051] At time slot t, the transmission delay of VNF between nodes is expressed as: Among them, d i (t) represents the data packet size of the VNF on the transmission path at time slot t, Indicates whether the t-time slot virtual link is mapped to the physical link. Indicates the bandwidth resources required by the t-time slot virtual link.

[0052] The propagation delay of a link can be expressed as: Among them D nm Is the physical link l nm distance, c(t) represents the speed of light.

[0053] The total link delay of VNF transmission from NFV node m to physical node n in time slot t can be expressed as:

[0054] VNF processing latency is the processing delay of node n in processing VNF in time slot t, expressed as:

[0055]

[0056] in Indicates the processing rate of the node for VNF, Indicates the data packet processing rate coefficient. Indicates whether the VNF in time slot t is mapped to physical node n.

[0057] In summary, the total delay on SFC can be expressed as:

[0058]

[0059] Among them, F represents the set of service function chains, is the set of VNFs on SFC i, is the set of virtual links on SFC i, Represents the virtual link between the j-th and k-th VNFs.

[0060] Then the average latency of all VNF instances running in time slot t can be expressed as:

[0061]

[0062] Among them, |N i | represents the number of VNFs on SFC i, so the average VNF delay within q time slots can be expressed as

[0063] Load balancing of physical nodes after VNF migration helps the network system be more robust in the face of traffic changes. Load balancing of physical nodes is an effective long-term utilization of network resources. When a VNF migrates from node n to node m at time slot t, this causes a load change on node m. This change is caused by the consumption of computing and storage resources on node m. The load on node m can be expressed as:

[0064]

[0065] in, and Represent the computing resource load and storage resource load of physical node m respectively, Indicates whether the VNF is mapped to the physical node m, They represent the computing and storage resources required by VNF respectively. The mean of computing resource load and storage resource load of physical node m are expressed as:

[0066]

[0067] Among them, C m and M m Represents the computing and storage resources owned by physical node m. The load balancing capability of the network system can be measured by the average network resource variance of the physical node. The average computing variance and average storage variance of the physical resources can be expressed as:

[0068]

[0069]

[0070] Load capacity of the network system L total (t) can be obtained by the weighted sum of the variance of computing and storage resources, which is expressed as follows:

[0071]

[0072] Among them, ω1 and ω2 represent the weight factors of computing resources and storage resources on the load balancing of the network system, respectively. Here, ω1+ω2=1. Then, the average total resource load variance within q time slots can be expressed as

[0073] S2: Under the multi-dimensional resource constraints of computing, memory, and bandwidth, we establish an optimization problem with the goal of minimizing the average VNF latency and average load resource variance. The specific problem is as follows:

[0074]

[0075] Where ξ1 and ξ2 are weight factors, τ i is the total delay threshold, x, y, Φ are A simplified representation of , where A binary variable indicating whether the VNF is mapped to a physical node. A binary variable indicating whether a virtual link is mapped to a physical link, A binary variable indicating whether the VNF is associated with a DT node; C n 、M n 、B n,m Respectively represent the computing resources, storage resources and bandwidth resources on the corresponding physical link owned by the physical node. These are the upper thresholds for computing resources, storage resources, and bandwidth resources, respectively.

[0076] The optimization objective is subject to constraints C1 to C7 to ensure its effectiveness: C1 ensures that each VNF can be mapped to only one physical node; C2 ensures that each virtual link can be mapped to only one physical link; C3-C4 represent the computing, storage, and bandwidth resource constraints of the physical node, respectively; C6 represents the SFC end-to-end delay constraint; and C7 is a series of binary variable constraints.

[0077] Furthermore, in step S2, VNF resource demand prediction is performed based on the resource usage data collected by the digital twin and the global SFC topology, as follows:

[0078] Design a digital twin-assisted prediction model (DT-TCN) based on graph convolutional neural network (GCN) and temporal convolutional network (TCN), such as Figure 2 As shown in Figure 2, this model fully extracts spatiotemporal features based on the global SFC topology of the digital twin to improve the accuracy of VNF resource demand prediction. The prediction model includes a graph learning module, a GCN network, and a TCN network.

[0079] The digital twin collects a large amount of data from the physical network layer in real time, including computing and storage on VNF nodes, as well as multi-dimensional resource information such as bandwidth on links. is the set of VNF nodes, and E is the set of virtual links. The adjacency matrix is ​​expressed as in The resource data of each VNF on the SFC collected by DT can be expressed as a multivariate time series, expressed as Where D represents the dimension of the feature. The multivariate time series with graph structure can be considered as a dynamic graph. At each time t, the resource usage of each VNF can be expressed as x t ={x1,x2,...,x n Resource prediction aims to predict the state of the graph at the next moment based on this historical information. This can be achieved by learning a function f(·) that maps the graph state at a given previous T time steps to the graph state at the next time step. Expressed as:

[0080] Graph Learning Module: Capturing spatial dependencies based on graph neural networks mostly relies on predefined graph structures. However, VNF deployment and the SFC graph structure dynamically change depending on the network environment. Therefore, it is necessary to mine inter-VNF node dependencies from the data and update the graph structure during training. This paper proposes a graph learning module that adaptively generates a graph adjacency matrix and fully utilizes the hidden relationships between VNFs, expressed as:

[0081] A=G gl (E) = Θ s (δ(M1M2 T -M2M1 T ))

[0082] in represents a learnable node embedding with random initialization, d is the node embedding dimension, and each element A i,j is the embedding E from the corresponding node i and E j recovered, It is obtained by multi-layer perceptron conversion, where M1 = tanh(ωE1σ1) and M2 = tanh(ωE2σ2), tanh(·) represents the tangent hyperbolic activation function, E1 and E2 represent node embeddings, σ1 and σ2 are model parameters, δ(·) = ReLU (tanh(ω(·)) is used to regularize the elements of the adjacent matrix, ω controls the saturation rate of the activation function, Θ s It is a row-wise sparse function. In addition, M1M2 T -M2M1 T To achieve the asymmetry of matrix A, so as to construct an adjacent matrix that is more in line with the dynamic changes of SFC topology in the actual network. gl (·) represents the graph learning function, through which the adjacency matrix is ​​obtained.

[0083] Time feature extraction module: In order to extract the long-term changing VNF operation data with short sampling intervals and large data volume, TCN design is used to extract the global information of the time series. Given a time series X = (x1, x2, ..., x t ) and convolution filter F=(f 1×1 ,f 1×2 ,...,f 1×l ), x t The dilated convolution at can be expressed as: in represents the convolution operation, d is the dilation factor that increases with the number of network layers, l is the filter kernel size, and the subscript x t-i·d Indicates the direction of the past.

[0084] By expanding the receptive field to extract long-term information, the expansion factor of each temporal convolution layer is set to increase exponentially at a rate of b (b>1). Given a receptive field w=1+d(l-1) and an initial expansion factor set to 1, the size of the expanded receptive field is expressed as: where b d Indicates the value of the dilation factor at the dth layer.

[0085] Spatial feature extraction module: After the graph learning module updates the adjacency matrix, the temporal features output by the temporal feature extraction module and the graph adjacency matrix reconstructed by graph learning are used as input, and the spatial propagation module based on GCN is used to extract spatial features. Let k represent the index of each spatial time layer. After reaching this module through each layer of the expanded convolution module, the neighbor node information is aggregated based on the data and the representation of the node data is updated: In the kth layer of the convolution module, the input after graph learning and temporal convolution is represented as H = Z (k) , then the layered propagation rule of graph convolution is expressed as: in Denotes the adjacency matrix of an undirected graph G with additional self-connections, I Nis the identity matrix, is the trainable weight matrix for a particular layer. is a diagonal matrix with self-loops, and are the node representations in the convolutional layers of the kth and k+1th graphs, respectively, and W (k) is the parameter matrix and σ is the nonlinear activation function.

[0086] In the final output layer, the skip connection layer is actually a 1×L k The standard convolutional layer, where L k Represents the input length of the k-th jump connection layer, normalizes the information that jumps to the output module so that they have the same output sequence length. The output module consists of two one-dimensional standard convolution modules, which convert the final prediction result Y i ={y1,y2,...,y n} to the desired output dimension sequence.

[0087] S3. For the proposed optimization problem P, the optimization problem is modeled as a Markov process, the state space, action and reward functions are defined, and a multi-agent based proximal optimization strategy is used to solve the optimal VNF digital twin deployment strategy. Figure 3 The following is a diagram of the framework of the VNF active migration algorithm based on DT-MASAC.

[0088] For the state space S, define s(t) = {ψ(t), ξ(t), C(t), M(t), B(t)}∈S to represent the network system state at time t, where ψ(t) is the network state information of the VNF node, ξ(t) is the network state information of the link, and C(t), M(t), and B(t) represent the state space of the VNF's computing, storage, and virtual link bandwidth resource requirements, respectively.

[0089] For the action space A, define a(t) = {a 1,1 (t),a 1,2 (t),...,a i,j (t),...}∈A represents the set of mapping actions that the VNF can take at time t, where a i,j (t) represents the mapping action of the j-th VNF on SFC i.

[0090] The transition probability p(s(t+1)|s(t),a(t)) represents the probability of transitioning to the next state s(t+1) after taking action a(t)∈A in the current state s(t)∈S.

[0091] When the network system takes action a(t) in state s(t), it will generate a mapping strategy π, which can obtain an instantaneous reward If the constraint is not satisfied, the reward r(t) = -1 / ε, where ε is the penalty factor.

[0092] Furthermore, in step S3, the DT-MASAC migration algorithm is specifically as follows:

[0093] The proposed DT-MASAC algorithm is an extension of the SAC algorithm in the multi-agent domain using a centralized training distributed execution framework. Each VNF is considered an agent. Each agent interacts with the VNF digital twin and obtains state information about the corresponding nodes and links from the digital twin environment, as well as information about the VNF's computing, storage, and bandwidth resource requirements for virtual links. The agent then takes the corresponding VNF mapping action and calculates the instantaneous reward. During the centralized training phase, the input to each agent's value network includes not only its own local state information, but also the local state information. i (t) and action information a i (t), also includes the state information of other agents o -i (t) and action information a -i (t). The policy network only needs to input the local state information of the agent to output the policy distribution, thereby obtaining actions through sampling. In the execution phase, each agent only needs to use the policy network to execute independently. The global state information is defined as s t ={o i (t),o -i (t)}, define the global action information as a t ={a i (t),a -i (t)}. By introducing the maximum entropy method, the exploration and stability of the algorithm are improved. The goal is to maximize the cumulative reward and policy entropy. The objective function is expressed as:

[0094]

[0095] where H(π(·|s t ))=-logπ(·|s t ) represents the policy entropy, π(·|s t ) represents the probability distribution of the mapping from state to action. α∈(0,1) represents the temperature coefficient, which determines the relative importance of the policy entropy term to the reward, thereby controlling the randomness of the optimal policy. π represents the state-action distribution under policy π.

[0096] The centralized Q value can be defined as:

[0097]

[0098] Where γ∈(0,1) is the discount factor, V(s t) is expressed as:

[0099] In the algorithm, each agent uses two value networks and The bias caused by overestimation of Q value can be reduced. The two value networks have the same network structure and do not share parameters. The value network is trained to minimize the soft Bellman residual, which can be expressed as:

[0100]

[0101] in represents the experience replay buffer, φ i,n is the parameter of the value network, which can be obtained by minimizing the loss function J Q (φ i,n ) to update, are the parameters of the target value network. The value network is trained by sampling experiences from the experience replay pool, and the target network can be periodically updated as Indicates that the policy network is in state s i,t+1 Next select action a i,t+1 The probability distribution of θ i are the parameters of the policy network.

[0102] The parameters θ of the policy network i It can be updated by minimizing. In order to facilitate the calculation of gradients, the reparameterization technique can be used, and the reparameterization technique can produce a lower variance estimate. i,t Reparameterized to where ∈ i,t is an input noise vector that can be sampled from some fixed distribution. Therefore, J π (θ i ) is expressed as:

[0103]

[0104] In the formula Represents the noise distribution, which is Gaussian distribution here. In addition, set a target policy entropy The temperature coefficient α is updated by minimizing the entropy loss function J(α):

[0105]

[0106] To summarize, the execution steps of the DT-MASAC algorithm are as follows:

[0107] 1) Calculate the resource utilization η of each VNF node based on the VNF resource demand prediction results R , determine whether the node is in an overloaded or underloaded state, and balance the node load by actively migrating VNFs;

[0108] 2) Initialize the Actor network parameters θ of each agent i , Critic network parameters and Experience Replay Pool

[0109] 3) The agent obtains network status information s i,t , and use the Actor network according to s i,t Get Action

[0110] 4) The agent gets the reward r for the corresponding VNF mapping and resource allocation based on the action i,t , and obtain the next local state information s i,t+1 ;

[0111] 5) Get global status information s t , global action information a t and the next global status information s t+1 ;

[0112] 6)Tuple(s t ,a t ,r i,t ,s t+1 ) stored in the experience replay pool from Randomly take small batches of experience data {(s j ,a j ,r i,j ,s j+1 )} as training data, and uses gradient descent to update the Actor network, Critic network, and temperature coefficients. The target value network is updated using a soft update method. This loop continues until the reward function converges or the maximum number of training rounds is reached.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A digital twin-assisted virtual network function active migration method based on resource prediction, characterized by: The method comprises the following steps: S1. Build a digital twin-assisted virtual network function migration architecture, which includes a physical network layer, a network slicing layer, and a digital twin layer. The physical network layer includes VNF nodes and digital twin service nodes. The VNF nodes are used to process VNF services, and the digital twin service nodes are used to complete real-time monitoring and data collection of VNFs. The network slicing layer deploys a series of SFCs on the VNF nodes using SDN / NFV technology to handle different network service requests. The digital twin layer builds a global network topology through the digital twin service nodes to achieve real-time monitoring and management of VNFs, while providing auxiliary verification for VNF resource demand prediction and migration strategy formulation. S2. Based on the collected information of the digital twin service nodes, a digital twin global network topology is constructed in the cloud. The spatiotemporal information of the network topology is extracted using a graph neural network to predict VNF ​​resource demand. Based on the prediction results, VNFs are actively migrated to nodes that are about to be overloaded or underloaded, so that the physical nodes maintain high performance and ensure the service QoS on the nodes. S3. Based on the VNF resource demand prediction results, a digital twin-assisted multi-agent actor-critic-based VNF active migration strategy is formulated, with the optimization problem established with the minimization of the VNF average latency and the average resource variance of the VNF nodes as the optimization objectives. S4. Convert the optimization problem into a Markov decision process, define the state space, action space, and reward function, and use a multi-agent-based proximal optimization strategy to solve the optimal VNF digital twin deployment strategy.

2. The method according to claim 1, wherein: In step S2, VNF resource requirements are predicted by designing a prediction model based on graph convolutional neural networks and temporal convolutional networks.

3. The method according to claim 2, wherein: The prediction model includes a graph learning module, a GCN network, and a TCN network; the graph learning module is used to adaptively generate a graph adjacency matrix; the TCN network extracts the temporal features of the VNF multi-resource data sequence through dilated temporal convolution; The temporal features output by the TCN network and the graph adjacency matrix reconstructed by the graph learning module are input into the GCN network to extract spatial features.

4. The method according to claim 3, wherein: The graph learning module uses the hidden relationships between VNFs to adaptively generate a graph adjacency matrix: A = G gl (E) = Θ s (δ(M1M2 T -M2M1 T )); where E represents a learnable node embedding with random initialization; M1 = tanh(ωE1σ1), M2 = tanh(ωE2σ2), where tanh(·) represents the tangent hyperbolic activation function, E1 and E2 represent node embeddings, σ1 and σ2 represent model parameters; δ(·) represents the element used to regularize the adjacency matrix; Θ s represents the row-wise sparse function; G gl (·) represents the graph learning function.

5. The method according to claim 1, wherein: In step S3, the optimization problem is expressed as: Where ξ1 and ξ2 represent weight factors, ξ1+ξ2=1, τ i Indicates the total delay threshold; and Represents the average VNF latency and average resource load variance in multiple time slots; A binary variable indicating whether the VNF is mapped to a physical node. A binary variable indicating whether a virtual link is mapped to a physical link, Binary variables indicating whether the VNF is associated with the DT node, x, y, and Φ are Simplified representation of; F represents the set of service function chains, represents the set of VNFs on SFC i, represents the set of virtual links on SFC i, represents the virtual link between the jth and kth VNFs; L P represents the set of all links between nodes, N P Represents a set of physical nodes, l nm Indicates a physical link; and They represent the computing resources and storage resources required by VNF respectively; C represents the bandwidth resources required by the t-time slot virtual link; n 、M n 、B n,m They represent the computing resources, storage resources and bandwidth resources on the corresponding physical links owned by the physical nodes. Respectively represent the upper thresholds of computing resources, storage resources, and bandwidth resources; Constraint C1 ensures that each VNF can only be mapped to one physical node; constraint C2 ensures that each virtual link can only be mapped to one physical link; constraints C3 to C5 represent the computing, storage, and bandwidth resource constraints of the physical node and the physical link, respectively; constraint C6 represents the SFC end-to-end delay limit; and constraint C7 represents a binary variable constraint.

6. The method according to claim 5, characterized in that: Convert the established optimization problem into a Markov decision process and define the state space, action space and reward function; For the state space S, let s(t) = {ψ(t), ξ(t), C(t), M(t), B(t)}∈S represent the network system state at time t, ψ(t) represents the network state information of the VNF node, ξ(t) represents the network state information of the link, C(t), M(t), and B(t) represent the state space of the VNF's computing, storage, and virtual link bandwidth resource requirements, respectively; For the action space A, define a(t)={a 1,1 (t),a 1,2 (t),...,a i,j (t),...}∈A represents the set of mapping actions that the VNF can take at time t, where a i,j (t) represents the mapping action of the j-th VNF on SFC i; For the reward function r(t), it is defined that the network system takes action a(t) in state s(t) to generate a mapping strategy π and obtain an instantaneous reward 7. The method according to claim 6, characterized in that: Based on the converted Markov process, a multi-agent-based proximal optimization strategy is used to solve the optimal VNF digital twin deployment strategy. The solution process includes: 1) Calculate the resource utilization η of each VNF node based on the VNF resource demand prediction results R , determine whether the node is in an overloaded or underloaded state, and balance the node load by actively migrating VNFs; 2) Initialize the Actor network parameters θ of each agent i , Critic network parameters and Experience Replay Pool 3) The agent obtains network status information s i,t , and use the Actor network according to s i,t Get Action Indicates that the policy network is in state s i,t Next select action a i,t The probability distribution of θ i are the parameters of the policy network; 4) The agent gets the reward r for the corresponding VNF mapping and resource allocation based on the action i,t , and obtain the next local state information s i,t+1 ; 5) Get global status information s t , global action information a t and the next global status information s t+1 ; 6)Tuple(s t ,a t ,r i,t ,s t+1 ) stored in the experience replay pool from Randomly take small batches of experience data {(s j ,a j ,r i,j ,s j+1 )} as training data, and use gradient descent to update the Actor network, Critic network and temperature coefficient, and update the target value network according to the soft update method; 7) The loop is executed until the reward function converges or the maximum number of training rounds is reached.

Images