Service function chain migration method and system based on user moving scene
Through the hierarchical reinforcement learning algorithm and graph convolution network to process the physical network topology, combined with the Transformer network to process migration request information, the global optimal problem of SFC migration in user mobile scenarios is solved, and the comprehensive minimization of migration costs, delays and end-to-end delays is achieved, which improves the decision quality and resource efficiency of SFC migration.
Patent Information
- Application Number
- CN202510769355.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-10
- Publication Date
- 2025-08-08
AI Technical Summary
In the highly dynamic user mobile scenario, existing SFC migration methods are difficult to find global optimal migration solutions, resulting in low network resource utilization and declining user experience. Traditional reinforcement learning methods face dimensional disasters and slow convergence speed when dealing with complex dynamic scenarios.
The hierarchical reinforcement learning algorithm is adopted, and the physical network topology is processed through the Actor-Critic framework and graph convolution network. The Transformer network processes migration request information, and the migration decision is decomposed into two levels of tasks: the target migration node and migration path are optimized. The migration cost, delay and end-to-end delay are used, and the context dependency is captured using the multi-head attention mechanism, and the reward function is designed to guide migration decisions.
It has achieved comprehensive minimization of migration costs, migration delays and end-to-end delays, improved the decision-making quality and resource efficiency of SFC migration, ensured that users enjoy a high-quality service experience, and avoided resource conflicts and service interruptions.
Smart Images

Figure CN120456127A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of service function chain migration, and in particular to a service function chain migration method and system based on user mobility scenarios. Background Art
[0002] With the rapid development of fifth-generation mobile communication technology (5G) and the widespread adoption of smart devices, mobile internet applications are experiencing explosive growth. Emerging services such as autonomous driving, virtual reality, and high-definition video streaming are placing unprecedented demands on network bandwidth, processing power, and latency. To address these challenges, mobile edge computing (MEC) has emerged. It shifts computing and storage resources to the network edge, closer to users, effectively reducing service latency and alleviating core network load. Service Function Chaining (SFC), a flexible network service delivery paradigm, allows multiple virtual network functions (VNFs) to be orchestrated in a specific order to meet diverse user needs. However, in highly dynamic user mobility scenarios, such as connected vehicle networks (VANETs) or when users switch between base stations, the existing SFC deployment may no longer be optimal and may even fail to guarantee service continuity and quality. Therefore, intelligently and efficiently migrating VNF instances from source nodes to more suitable destination nodes based on user mobility and network state changes to ensure quality of service (QoS) and optimize network resource utilization has become a critical issue in current network technology, crucial for improving user experience and network operational efficiency. Currently, traditional SFC migration methods exhibit numerous shortcomings when addressing complex and dynamic user mobility scenarios. While many strategies based on heuristic or greedy algorithms offer rapid decision-making, they often struggle to find the globally optimal migration solution and are prone to falling into local optima, resulting in low network resource utilization and a degraded user experience. Furthermore, some early reinforcement learning methods often suffer from the curse of dimensionality and slow convergence due to the large state and action spaces involved in SFC migration. Furthermore, they lack the ability to perceive the dynamic network topology and capture the complex dependencies between VNFs within the SFC, resulting in limited decision-making accuracy and adaptability. Summary of the Invention
[0003] The purpose of the present invention is to provide a service function chain migration method and system based on user mobility scenarios, which achieves the comprehensive minimization of migration cost, migration delay and subsequent end-to-end delay, and significantly improves the decision-making quality, resource efficiency and user experience of SFC migration.
[0004] The present invention is achieved through the following technical solutions: A method for migrating a service function chain based on a user mobility scenario, the method comprising the following steps: Responding to the service function chain migration demand caused by user mobility, obtaining migration request information of the service function chain to be migrated, which includes the resource requirements, performance indicators and current physical network resource status information of each virtual network function in the service function chain to be migrated; Determine the migration decision task for each virtual network function in each service function chain to be migrated. The migration decision task includes: the target migration node, the bandwidth resources required for migration, and the specific migration path from the current node to the target migration node; On the premise of meeting the resource requirements and performance indicators of the service function chain, the comprehensive minimization of migration cost, migration delay and end-to-end delay of the service function chain after migration is defined as the optimization goal, the determined migration decision task is optimized and calculated, and the optimal service function chain migration decision plan is output.
[0005] Optionally, the optimization decision process for each virtual network function in each service function chain to be migrated is determined by a hierarchical reinforcement learning algorithm, specifically as follows: Decompose the migration decision task of the service function chain into two levels of subtasks: determining the target migration node of the virtual network function and determining the virtual network function migration path; A two-layer Actor-Critic reinforcement learning framework is established, in which the upper-layer reinforcement learning agent is responsible for deciding the target migration node of the virtual network function, and the lower-layer reinforcement learning agent is responsible for deciding the migration path of the virtual network function.
[0006] Optionally, before executing the decision, the hierarchical reinforcement learning algorithm further includes: Enter the resource status information of the current physical network; Processing the topological structure and node and link features of the physical network through a graph convolutional network to extract representation information of the physical network; The physical network representation information is provided as input to the upper-layer reinforcement learning agent and the lower-layer reinforcement learning agent.
[0007] Optionally, before executing the decision, the hierarchical reinforcement learning algorithm further includes: Processing migration request information of the service function chain to be migrated through the attention mechanism of the Transformer network; The service function chain migration request information is input into the encoder network, and the contextual dependencies and virtual network function features in the service function chain sequence are captured from different dimensional spaces through a multi-head attention mechanism; The extracted virtual network function features are provided as input to the upper-layer reinforcement learning agent and the lower-layer reinforcement learning agent.
[0008] Optionally, the state input of the upper-layer reinforcement learning agent includes: physical network representation information and virtual network function characteristics; the upper-layer reinforcement learning agent outputs the decision of the target migration node of the virtual network function, and the calculation formula of the upper-layer reinforcement learning agent is:
[0009]
[0010]
[0011] in, is the adjacency matrix of the physical network, is the characteristic matrix of the physical node, is a column vector, is the number of virtual network functions in the service function chain migration request, is the state input to the agent, It is the physical network topology status. Migrate request status for service function chain.
[0012] Optionally, the state input of the lower-layer reinforcement learning agent includes: physical network representation information, virtual network function characteristics, and the virtual network function target migration node output by the upper-layer reinforcement learning agent; the lower-layer reinforcement learning agent outputs the decision of the virtual network function migration path, and the calculation formula of the lower-layer reinforcement learning agent is:
[0013]
[0014] in, It is the characterization information of the physical network, Consistent, is a column vector.
[0015] Optionally, the upper-layer reinforcement learning agent and the lower-layer reinforcement learning agent are further provided with a reward function, wherein the reward function of the upper-layer reinforcement learning agent is:
[0016] in, 、 and are weights, is the discount factor for the reward, 、 are the migration costs, The end-to-end latency of the service function chain after migration; The reward function of the lower-level reinforcement learning agent is:
[0017] in, and is the weight size, is the migration cost, Delayed migration.
[0018] Optionally, the optimization objective is to define the comprehensive minimization of migration cost, migration delay, and end-to-end delay of the service function chain after migration, and the calculation formula is:
[0019] in, 、 、 、 are coefficients, is a binary variable used to indicate whether the VNF is deployed on node u at time t. If yes, it is 1, otherwise it is 0. is the cost of instantiating VNF on the node, V is the node set, F is the VNF set, R is the request set, 、 are the maximum delay of migration request and the maximum end-to-end delay of request, The kth request in the request set.
[0020] Optionally, the optimization objective is also set with multiple constraints, including: instantiated VNF quantity constraint, VNF single-node deployment constraint, VNF instantiation cost judgment constraint, resource limitation constraint, source-destination node constraint, flow conservation constraint, migration flow conservation constraint, migration delay and SFC end-to-end delay constraint.
[0021] A service function chain migration system based on user mobility scenarios, comprising: An acquisition unit responds to a service function chain migration demand caused by user mobility and acquires migration request information of the service function chain to be migrated, where the migration request information includes resource requirements, performance indicators, and resource status information of the current physical network for each virtual network function in the service function chain to be migrated; A decision-making unit determines a migration decision task for each virtual network function in each service function chain to be migrated. The migration decision task includes: the target migration node, the bandwidth resources required for migration, and the specific migration path from the current node to the target migration node; The optimization unit defines the comprehensive minimization of migration cost, migration delay and end-to-end delay of the service function chain after migration as the optimization goal, on the premise of meeting the resource requirements and performance indicators of the service function chain, optimizes the calculation of the determined migration decision task, and outputs the optimal service function chain migration decision plan.
[0022] The technical solution of the present invention has at least the following advantages and beneficial effects: The present invention provides significant technical advantages by responding to the service function chain migration requirements triggered by user mobility and globally optimizing migration decisions. First, by comprehensively obtaining the resource requirements, performance indicators, and real-time resource status information of the service function chain to be migrated, including each virtual network function, the present invention enables migration decisions to be based on more comprehensive and accurate data, thereby significantly improving the feasibility and effectiveness of the migration plan and avoiding blind migration or resource conflicts caused by insufficient information. Second, the present invention independently determines the target migration node, required bandwidth, and specific migration path for each virtual network function in each service function chain to be migrated, making the optimization calculation more accurate and able to specifically meet the needs of each VNF, thereby ensuring the performance of the entire service function chain during and after the migration process. The core advantage is that the present invention takes the comprehensive minimization of migration cost, migration delay, and end-to-end delay of the service function chain after migration as the optimization goal, and outputs the optimal migration decision plan through optimization calculation, ensuring that, under the premise of meeting the resource requirements and performance indicators of the service function chain, the operator's operating costs can be reduced to the greatest extent, the service interruption or jitter time perceived by users can be shortened, and users can enjoy a high-quality service experience after migration. Compared with traditional technologies that often focus on one thing and neglect another or can only achieve local optimization, the present invention realizes the coordinated optimization and global optimization of multi-dimensional performance indicators, significantly improving the intelligence level and comprehensive benefits of SFC migration in dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A flow chart of a service function chain migration method based on user mobility scenarios provided by the present invention; Figure 2 This is an example diagram of SFC migration in the user mobility scenario provided by the present invention; Figure 3 The SFCM network structure diagram provided by the present invention; Figure 4 A schematic diagram of the hierarchical reinforcement learning structure provided by the present invention; Figure 5 Flowchart of the SFC migration algorithm based on hierarchical reinforcement learning provided by the present invention. DETAILED DESCRIPTION
[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.
[0025] like Figure 1 、 Figure 2 As shown, the present invention provides one embodiment: a service function chain migration method based on a user mobility scenario, the specific steps of which are as follows: S1. In a static scenario, this embodiment first models the service function chain migration problem as an integer linear programming model, and jointly optimizes the migration cost, delay, and end-to-end delay of the function chain; S2. Due to the complexity of the migration problem, the ILP method cannot meet the reliability requirements of the SFC migration problem in dynamic scenarios. In this case, the present invention decomposes the complex task of SFC migration into two subtasks, introduces a hierarchical structure, and designs a service function chain migration method based on hierarchical reinforcement learning. This method can more efficiently solve complex high-dimensional problems and is suitable for SFC migration optimization in large-scale networks. In step S1, considering the constraints of network resource consumption, migration delay, and end-to-end delay in the migration problem, this embodiment designs a cost-effective and delay-aware SFC migration method: S11. Establish the optimization objective function, expressed as:
[0026] The objective function consists of four parts, the first two of which represent the activation cost and VNF instantiation cost respectively. Is a binary number indicating whether the server is alive, then the first part Indicates the cost of activating the server; in the second part represents the cost of instantiating the VNF on the node, so the second part Refers to the VNF instantiation cost at the current moment; the third and fourth parts represent the average migration delay of VNF in SFC and SFC end-to-end delay, respectively. 、 Represents the maximum delay of migration request and the maximum end-to-end delay of request respectively; coefficient 、 、 To quantify the linear sum of the three; S12. Limit the number of instantiated VNFs in each SFC to between the total number of VNFs and the number of VNF deployment nodes, expressed as:
[0027] in is a binary variable, indicating Whether the VNF is instantiated on node u at this moment, It is also a binary variable, indicating that One request at a time Whether the VNF in is deployed on node u, R indicates multiple A collection of multiple service function chain requests that need to be migrated. represents the jth VNF in the SFC, and F represents the set of VNFs in multiple SFCs; S13. Each VNF in an SFC is restricted to being deployed on a single node. A single VNF cannot be deployed across nodes. If a VNF is successfully deployed on a node, the node is activated and the corresponding cost is incurred, expressed as:
[0028] in Is a binary variable, indicating a request at time t Whether the VNF in is deployed on node u, Express a request Required service function chain; S14. Determine whether VNF needs to be spent at time t The instantiation cost on , is expressed as:
[0029] Only when There is no VNF on node u at this moment When the instance is, at the next time t, VNF A new instance needs to be instantiated on the node to incur the instantiation cost; S15, resource limitation constraints, due to the limited resources of nodes and links in the network, it is necessary to request The VNF computing resource requirements and link bandwidth resource requirements in the SFC are restricted, which can be expressed as: ,in, Express a request traffic demand, represents the resource capacity of node u. This formula is used to constrain the total computing resources consumed by the VNF to not exceed the resource capacity of node u. ,in, Is a binary variable, indicating the physical link Is there a request? SFC link used to connect VNF With VNF , represents the link bandwidth capacity between nodes u and v. This formula is used to constrain the link bandwidth consumption between VNFs not to exceed the physical link bandwidth. Bandwidth resource capacity on S16, source and destination node constraints, expressed as:
[0030]
[0031] in, is a binary variable, ensuring that VNF Can only be placed on the source node; is a binary variable, ensuring that VNF It can only be placed on the destination node; S17, flow conservation constraint, expressed as:
[0032] Indicates the adjacent VNFs on a service function chain With VNF If they are all deployed on node u, no link is needed. In addition, there is at least one link connecting the VNFs. With VNF , and the traffic flows to node u or v, where Is a binary variable, indicating the physical link Is there a request? SFC link used to connect VNF With VNF ; S18, the migration flow conservation constraint is expressed as:
[0033] Indicates that when a VNF on an SFC is migrated, there is at least one migration path connection. The original node where the VNF is deployed at time t and the current node to which the VNF needs to be migrated at time t, where Is a binary variable, indicating the physical link Is it in VNF in the migration path; S19, migration delay and SFC end-to-end delay constraints are expressed as:
[0034] in, Indicates a physical link The migration delay, The maximum migration delay tolerated by users. This formula is used to constrain the SFC migration delay to not exceed the maximum migration delay.
[0035] in, Represents the maximum value of the end-to-end delay of the service function chain SFC. This formula is used to constrain the end-to-end delay of the SFC after migration to not exceed the maximum service delay. Since the integer linear programming problem in step S1 is not suitable for a highly dynamic network environment, the present invention decomposes the complex task of SFC migration into two subtasks in a dynamic scenario, introduces a hierarchical structure, and designs a service function chain migration method based on hierarchical reinforcement learning to more efficiently solve complex problems of high dimensions. It is characterized in that: based on hierarchical reinforcement learning, a two-layer Actor-Critic framework is established to decompose the complex problem of SFC migration into two subtasks, and two layers of reinforcement learning agents respectively decide the VNF migration destination node and the VNF migration path, and jointly optimize the cost, delay and end-to-end delay problems in the migration. Both the upper and lower layers of reinforcement learning include the underlying physical network and the SFC migration request feature extraction network, and use the characteristics of sequential decision-making in the Markov decision process to simulate the random strategies and rewards in an environment where the system state has Markov properties.
[0036] S21. A two-layer actor-critic deep hierarchical reinforcement learning algorithm based on Markov decision-making is established. This algorithm consists of an upper-layer agent and a lower-layer agent. These two layers of agents handle finding the VNF migration destination node and VNF migration path in the SFC migration problem, respectively. The algorithm also leverages the sequential decision-making characteristics of the Markov decision process to simulate random strategies and rewards in an environment where the system state exhibits Markov properties. S22. Use a graph convolutional network (GCN) to perform convolution operations on the graph to extract node resource capacity and link bandwidth capacity of the physical network based on neighbor node and link features; S23. Use the Transformer attention mechanism to process the SFC migration request information features. Input the SFC migration request information features into the encoder network and use the multi-head attention mechanism to process the SFC migration request information features from different dimensional spaces, while effectively capturing the contextual information in the SFC sequence. S24. After passing the physical network and SFC requests through the feature extraction network, they pass through two fully connected layers to obtain a probability distribution matrix for candidate actions. The dimension of the vector finally output by the upper-layer agent is the number of destination nodes for VNF migration in the SFC, and the dimension of the vector finally output by the lower-layer agent is the number of VNF migration paths in the SFC. The two-layer agent samples actions from the probability distribution matrix. S25: After the upper-layer agent executes the action, the current network state and the migration SFC request state change. When a migration SFC request is received, the upper-layer agent receives the upper-layer reward given by the environment. At the same time, the upper-layer action and the current network state are transmitted to the lower-layer agent. The lower-layer agent repeats steps S22-S24, executes the lower-layer action, and changes the current state. When a migration SFC request is received, the lower-layer agent also receives the lower-layer reward. S26, the corresponding Actor-Critic layer calculates the loss function and updates the corresponding network parameters; S27. Repeat steps S21-S26 to calculate the corresponding cumulative rewards until the model converges or the iteration number is reached and the iteration is stopped.
[0037] Furthermore, in step S21, this embodiment transforms the SFC migration problem into a two-layer agent sequential decision problem. The two-layer agent decomposes the SFC migration problem into two different subtasks and transforms the subtasks into an MDP model. The state transition is defined as ,in Indicates the current state, Indicates the current action. Represents the reward given by the environment after executing the action, Indicates the next state after executing the current action. Since this embodiment involves a hierarchical reinforcement learning framework, the states corresponding to the upper and lower intelligent agents are different. In the upper intelligent agent, the state is defined as ,in , , Represents the state input to the agent, including and , Indicates the status of the physical network topology. Indicates the status of the service function chain migration request, specifically, the physical network topology status After being processed by the graph convolutional network, the request status is migrated with the service function chain After being processed by the Transformer encoder, the resulting feature representations will be used as key inputs and fed into the subsequent SFCM network structure for deeper feature fusion. represents the adjacency matrix of the underlying physical network, Represents the feature matrix of the physical node, Each row of is a four-dimensional feature vector, including the total capacity and remaining capacity of the computing resources of the physical node, as well as the total bandwidth resources and available remaining bandwidth resources of the adjacent edges of the node. in is a column vector, representing the four-dimensional feature vector of VNFi, including the amount of computing resources consumed by the VNF, the type of the VNF, the traffic volume of the SFC, and the number of VNFs in the SFC that have not found the migration destination node. Indicates the number of VNFs in the SFC migration request. In the lower-level agent, the state is defined as , because the two agents share the same underlying network, the first part As in the first part of the upper-level agent, the distinction between subscripts r and k ensures clarity in defining the input and functionality of each level in the hierarchical structure, thus facilitating a structured solution to the complex SFC migration problem. The second part in is a column vector, representing the dirty page rate, memory size, source node and destination node of VNFi, Indicates the number of VNFs in the SFC migration request.
[0038] In a further implementation of this embodiment, in step S22, since there is a certain important relationship between the network nodes and the surrounding neighboring nodes in the network topology, the characteristics of the surrounding neighboring nodes such as the computing resource capacity and the link bandwidth capacity between them will affect the migration and migration path planning performance of the VNF, and the physical network topology graph structure involved in this embodiment is a non-Euclidean structure, so this embodiment uses a graph convolutional neural network GCN to capture the node and link feature information in the network. First, the current physical network state Input GCN layer, and get the new feature matrix after GCN processing . As shown below: ,in, is the trainable parameter matrix, is the activation function, is the degree matrix, is the sum of the unit matrix and the adjacency matrix of the underlying network topology. In steps S23-S24, in order to capture the global information features of the SFC migration request and ensure the sequence relationship of the VNF, this embodiment inputs the SFC migration request into the encoder network and uses a multi-head attention mechanism to process the feature information of the SFC migration request from different dimensional spaces. SFC migration request feature extraction: First, an SFC migration request feature matrix After the position encoding structure, it ensures the order requirements of SFC migration requests, and then extracts global feature information through the multi-head attention mechanism, and finally outputs the feature After layer normalization and feedforward neural network dimension transformation, As a query vector Input to the decoder's second multi-head attention module.
[0039] Physical network feature extraction: In order to further process the feature information of the network topology, the present invention extracts the physical network feature information after GCN processing. Input to the attention mechanism decoder network, the attention mechanism includes a multi-layer multi-head attention mechanism, which can better obtain the global information between nodes and nodes, and nodes and links. ,key Sum The calculation is as follows:
[0040] in, is a learnable weight parameter matrix. To calculate the matching degree between SFC and physical network topology, the present invention uses the output of the multi-head attention mechanism of the first layer of the decoder as the key Sum , the final output after SFC migration request feature extraction As a query vector , input the second layer multi-head attention mechanism of the decoder, and then process the feature information through the feedforward neural network to output the VNF migration strategy. In step S24, since the present invention decomposes the VNF migration problem into two subtasks, the action definition and action space of the upper and lower intelligent agents are different. The upper action space , whose dimension is the number of nodes , an action in the action space It is defined as selecting a node from the node set as the migration destination node of the VNF. is the number of VNFs in SFC. Lower action space , whose dimension is the number of optional paths for VNF migration , an action in the action space Defined as selecting a VNF migration path from a set of migration optional paths, dimension is the number of VNFs in the SFC.
[0041] In order to obtain the probability distribution of candidate actions, the present invention connects a feedforward neural network after the second layer multi-head attention mechanism of the decoder, which is a fully connected layer with shared parameters, transforming the feature vector dimension into the action space dimension or , and finally use the activation function A probability distribution matrix is obtained, and the upper-layer or lower-layer agents can obtain corresponding actions through random sampling. In step S25, this embodiment designs a reward function based on the cost, delay, and end-to-end latency of the migration. The migration cost is divided into three aspects. The first is the cost of computing resources consumed by all VNFs on the SFC, expressed as: ,in, Is a binary variable, indicating the migration request in Whether it is newly deployed on node u, Indicates that a new deployment is made on node u On the other hand, the migration cost also includes the bandwidth consumption cost of all migration requests passing through the link, which is expressed as: , where the binary number Indicates a physical link Is there a request? SFC link, used to connect and The third is the migration bandwidth cost allocated to the VNF, expressed as: , where the binary number Indicates the migration request Whether migration occurs, Indicates that the Migration bandwidth. Migration delay is the user downtime caused by the migration of the VNF on the migration request. If the migration delay is too large, it will greatly affect the service quality. It is expressed as: ,in, Indicates the migration request The time it takes to migrate to the destination node. Typically, a service requested for migration cannot be restarted until all VNFs have migrated. Therefore, the migration delay is the maximum of all VNF migration times on the SFC. End-to-end latency refers to the end-to-end latency of the service function chain after the migration is complete. After the migration is complete, the SFC is restarted and continues to be used. A small end-to-end latency significantly improves user service quality. Therefore, the migration decision also needs to consider the end-to-end latency of the service function chain after the migration, expressed as: ,in, Indicates a link The upper-level agent makes a decision to select the migration destination node, so the reward function of the upper-level decision is set for: ,in 、 and is the weight size, The reward function in this embodiment is designed to evaluate the quality of the upper-level agent's selection of migration destination nodes for each VNF in an SFC. Its core function is to first determine the feasibility of the upper-level agent's decision, specifically whether each selected migration destination node can meet the resource requirements of its corresponding VNF. The specific reward calculation logic is as follows: After the upper-level agent has selected migration destination nodes for all VNFs in a specific SFC, the system performs a verification process. This verification process confirms whether each selected migration destination physical node has sufficient resources to meet the preset resource requirements of the VNF it will host. Based on the feasibility determination results, the reward is assigned as follows: Case 1: All migration destination nodes meet the resource requirements. If the verification confirms that the migration destination nodes selected by the upper-level agent for all VNFs in the SFC can meet the resource requirements of their respective VNFs, meaning that all VNFs are successfully accepted by the target nodes, the decision is deemed feasible from a resource perspective. At this point, the upper-level agent receives a quantitative reward calculated according to predefined rules. The reward function in this embodiment is designed to reward the upper-level agent more when the overall cost of a migration decision is low, provided that the decision is resource-feasible (i.e., meets the acceptance criteria). This encourages the agent to learn and select migration solutions that are not only resource-feasible but also offer better overall performance and lower costs. Case 2: At least one migration target node fails to meet resource requirements. If verification shows that at least one of the migration target nodes selected by the upper-level agent for the SFC fails to meet the resource requirements of the VNF it intends to host—that is, at least one VNF fails to be successfully accepted by its designated migration target node—this indicates that the decision is resource-infeasible. In this case, the upper-level agent receives a clear negative reward. Negative rewards are intended to strongly prevent the agent from making invalid decisions that do not meet basic resource constraints. The upper-level agent considers factors such as the node resource consumption cost after migration, the link bandwidth consumption cost, and the end-to-end delay size to select the best destination node for each VNF in the migration request. It then confirms whether the destination node meets the VNF computing resource requirements and whether the link meets the virtual link bandwidth requirements included in the SFC. If so, a reward is given, otherwise a negative reward is given. Based on the decision of the lower-level agent to select the migration path, the reward function of the lower-level decision is set. for: ,in and Similar to the reward mechanism for the upper-level agent, the reward function for the lower-level agent also first determines the feasibility of its decision. The core criterion is whether the planned migration path can meet the bandwidth requirements of the VNF migration process, that is, whether the migration link can accommodate the data traffic during the VNF migration. The specific reward calculation logic is as follows: After the lower-level agent plans a migration path from the source node to the target node for a VNF, the system verifies whether the available bandwidth along the path meets the bandwidth required for the VNF migration. Based on the feasibility determination, the reward is assigned as follows: Case 1: The migration path meets the bandwidth requirements. If the migration path planned by the lower-level agent for the VNF is verified to meet the bandwidth requirements for migration, that is, the migration link can successfully accommodate the VNF migration traffic, then the migration path is considered feasible in terms of bandwidth. At this point, the lower-level agent receives a quantitative reward calculated according to pre-defined rules. When a migration path decision is feasible within bandwidth (i.e., meets acceptance criteria) and has a low associated overall cost, the lower-level agent receives a higher reward; otherwise, it receives a lower reward. This incentivizes the agent to learn and select migration paths that are not only bandwidth-consistent but also have higher overall transmission efficiency and lower resource consumption. Case 2: Migration path fails to meet bandwidth requirements. If verification shows that the migration path planned by the lower-level agent for the VNF does not meet the bandwidth requirements for migration, meaning that the migration link fails to successfully accept the VNF's migration traffic, this indicates that the path planning decision is bandwidth-infeasible. In this case, the lower-level agent receives a clear negative reward, or penalty. Through this mechanism, the lower-level agent is guided to learn and select migration paths that first meet the VNF migration bandwidth requirements and then further optimize the path-related costs and latency. The lower-level agent selects the optimal migration path for each VNF, taking into account migration bandwidth cost and latency. Each link along the migration path is sequentially determined to determine whether it meets the VNF migration bandwidth requirements. If so, a reward is awarded; otherwise, a negative reward is given. The present invention adopts a two-layer Advance Actor-Critic (A2C) to train the neural network using the proximity policy optimization (PPO). Both the policy network and the value network use the SFCM network structure as shown in the figure. The policy network outputs the SFCM network structure through a fully connected layer to obtain the action probability matrix, while the value network outputs the current state value through a single neuron.
[0042] like Figure 3 、 Figure 4 As shown, step S2 includes a graph convolutional neural network and a multi-head attention mechanism for feature extraction, hierarchical reinforcement learning, and the interaction between the two-layer agent and the environment. The neural network is trained using two layers of A2C using neighboring policy optimization (PPO). At time t, the upper-layer agent inputs the state into the SFCM network structure, samples an upper-layer action from the action probability matrix, and interacts with the environment to obtain the next state. and rewards Then the lower-level agent selects a lower-level action based on the upper-level action and the current state, and interacts with the internal drive to obtain the next state. and rewards , the internal drive responds to the environment, and the two-layer agent uses the neighboring policy optimization (PPO) to update the corresponding policy network and value network. , first use the value network to calculate the time series difference error : ,in, is the discount factor, is the observed reward, is the value network's estimate of the state value at time t+1. Then recursively calculate the advantage function, which represents the value of the state at time t+1. Next, action The deviation from the mean, the advantage function is calculated using GAE: ,in, is a hyperparameter of GAE that controls the trade-off between bias and variance in the estimate.
[0043] The present invention uses PPO-Clip to directly perform a certain degree of clipping on the ratio of the new and old strategies to constrain the range of change. The objective function is calculated as follows:
[0044] in, Calculate the ratio of probabilities to control the extent of new strategy updates. Represents the left side of the proximal strategy optimization pruning objective function formula, The policy network parameters are updated by calculating the loss of this policy update, and the value network loss is calculated based on the gap between the estimated value and the target value, thereby updating the parameters.
[0045] like Figure 5 As shown, this embodiment first initializes the states of the upper and lower layers and the corresponding policy network and value network parameters. The upper-layer agent inputs the state into the SFCM network and samples an upper-layer action. It selects a destination node for each VNF in the SFC and determines whether the destination node meets the resource constraints. After entering the next state, it obtains an upper-layer (external) reward from the environment. The lower-layer agent performs a lower-layer action based on the current upper-layer action and state to select a migration path for each VNF, determines whether the migration link meets the resource constraints, and obtains a lower-layer (internal) reward from the internal driver after entering the next state. The internal driver responds to the environment. The two-layer agents use the neighboring policy optimization (PPO) to train the neural network in turn to make the best decision for SFC migration.
[0046] To verify the effectiveness of this embodiment in solving the SFC migration problem in user mobility scenarios, we selected current mainstream methods for comparison, including the DDPG algorithm, the DQN algorithm, and the TD3 algorithm. All algorithms used the same environment settings and experimental parameters. To evaluate the performance of this invention, we used migration cost, migration delay, and end-to-end latency of the functional chain as evaluation metrics for the comparative experiments.
[0047]
[0048] Table 1 above quantifies the performance comparison of various methods. The proposed method outperforms the comparison methods in terms of migration cost, migration delay, and end-to-end latency of the functional chain. Therefore, in terms of comprehensive model performance, the proposed method performs best.
[0049] The above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.
Claims
1. A service function chain migration method based on user mobility scenarios, characterized in that: The steps of the method include: Responding to the service function chain migration demand caused by user mobility, obtaining migration request information of the service function chain to be migrated, which includes the resource requirements, performance indicators and current physical network resource status information of each virtual network function in the service function chain to be migrated; Determine the migration decision task for each virtual network function in each service function chain to be migrated. The migration decision task includes: the target migration node, the bandwidth resources required for migration, and the specific migration path from the current node to the target migration node; On the premise of meeting the resource requirements and performance indicators of the service function chain, the comprehensive minimization of migration cost, migration delay and end-to-end delay of the service function chain after migration is defined as the optimization goal, the determined migration decision task is optimized and calculated, and the optimal service function chain migration decision plan is output.
2. The service function chain migration method based on user mobility scenario according to claim 1, characterized in that: The optimization decision process for each virtual network function in each service function chain to be migrated is implemented through a hierarchical reinforcement learning algorithm, specifically as follows: Decompose the migration decision task of the service function chain into two levels of subtasks: determining the target migration node of the virtual network function and determining the virtual network function migration path; A two-layer Actor-Critic reinforcement learning framework is established, in which the upper-layer reinforcement learning agent is responsible for deciding the target migration node of the virtual network function, and the lower-layer reinforcement learning agent is responsible for deciding the migration path of the virtual network function.
3. The service function chain migration method based on user mobility scenario according to claim 2, characterized in that: The hierarchical reinforcement learning algorithm further includes: Enter the resource status information of the current physical network; Processing the topological structure and node and link features of the physical network through a graph convolutional network to extract representation information of the physical network; The physical network representation information is provided as input to the upper-layer reinforcement learning agent and the lower-layer reinforcement learning agent.
4. The service function chain migration method based on user mobility scenario according to claim 3 is characterized in that: The hierarchical reinforcement learning algorithm further includes the following steps before executing the decision: Process the migration request information of the service function chain to be migrated through the attention mechanism of the Transformer network; The service function chain migration request information is input into the encoder network, and the contextual dependencies and virtual network function features in the service function chain sequence are captured from different dimensional spaces through a multi-head attention mechanism; The extracted virtual network function features are provided as input to the upper-layer reinforcement learning agent and the lower-layer reinforcement learning agent.
5. The service function chain migration method based on user mobility scenario according to claim 4 is characterized in that: The upper-layer reinforcement learning agent, whose state input includes: physical network representation information and virtual network function characteristics; the upper-layer reinforcement learning agent outputs the decision of the target migration node of the virtual network function, and the calculation formula of the upper-layer reinforcement learning agent is: in, is the adjacency matrix of the physical network, is the characteristic matrix of the physical node, is a column vector, is the number of virtual network functions in the service function chain migration request, is the state input to the agent, It is the physical network topology status. Migrate request status for service function chain.
6. The service function chain migration method based on user mobility scenario according to claim 5, characterized in that: The state input of the lower-layer reinforcement learning agent includes: physical network representation information, virtual network function characteristics, and the virtual network function target migration node output by the upper-layer reinforcement learning agent; the lower-layer reinforcement learning agent outputs the decision of the virtual network function migration path, and the calculation formula of the lower-layer reinforcement learning agent is: in, It is the characterization information of the physical network, Consistent, is a column vector.
7. The service function chain migration method based on user mobility scenario according to claim 6, characterized in that: The upper layer reinforcement learning agent and the lower layer reinforcement learning agent are further provided with a reward function, wherein the reward function of the upper layer reinforcement learning agent is: in, 、 and are weights, is the discount factor for the reward, 、 are the migration costs, The end-to-end latency of the service function chain after migration; The reward function of the lower-level reinforcement learning agent is: in, and is the weight size, is the migration cost, Delayed migration.
8. The service function chain migration method based on user mobility scenario according to claim 7, characterized in that: The optimization goal is to minimize the migration cost, migration delay, and end-to-end delay of the service function chain after migration. The calculation formula is: in, 、 、 、 are coefficients, is a binary variable used to indicate whether the VNF is deployed on node u at time t. If yes, it is 1, otherwise it is 0. is the cost of instantiating VNF on the node, V is the node set, F is the VNF set, R is the request set, 、 are the maximum delay of migration request and the maximum end-to-end delay of request, The kth request in the request set.
9. The service function chain migration method based on user mobility scenario according to claim 8, characterized in that: The optimization objective is also subject to multiple constraints, including: instantiated VNF quantity constraint, VNF single-node deployment constraint, VNF instantiation cost judgment constraint, resource limitation constraint, source-destination node constraint, flow conservation constraint, migration flow conservation constraint, migration delay and SFC end-to-end delay constraint.
10. A service function chain migration system based on user mobility scenarios, characterized in that: include: An acquisition unit responds to a service function chain migration demand caused by user mobility and acquires migration request information of the service function chain to be migrated, where the migration request information includes resource requirements, performance indicators, and resource status information of the current physical network for each virtual network function in the service function chain to be migrated; A decision-making unit determines a migration decision task for each virtual network function in each service function chain to be migrated. The migration decision task includes: the target migration node, the bandwidth resources required for migration, and the specific migration path from the current node to the target migration node; The optimization unit defines the comprehensive minimization of migration cost, migration delay and end-to-end delay of the service function chain after migration as the optimization goal, on the premise of meeting the resource requirements and performance indicators of the service function chain, optimizes the calculation of the determined migration decision task, and outputs the optimal service function chain migration decision plan.
Citation Information
Cited By
Service traffic migration method, computer equipment, medium and program product
CN121644640A
