A dynamic orchestration method for NFV service function chain based on federated attention reinforcement learning
Through the federal attention reinforcement learning method, the placement scheme of virtual network function sets is coordinated to train, and the problem of slow dynamic orchestration response in the existing technology is solved, efficient service function chain orchestration and resource utilization is achieved, cost reduction and data protection privacy.
Patent Information
- Application Number
- CN202211060108.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-31
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2042-08-31
AI Technical Summary
Existing SFC orchestration methods, especially dynamic orchestration schemes, respond slowly, find it difficult to effectively utilize network resources, and is difficult to train in large-scale state spaces, resulting in inefficiency when meeting long-term needs.
Using a method based on federated attention reinforcement learning, through collaborative training between cloud and local clients, a virtual network function set placement scheme is generated, and the global model is dynamically optimized by attention neural network and DQN algorithm to realize flexible orchestration of the service function chain and reduce the communication costs and orchestration efficiency are improved.
It realizes dynamic orchestration of the service function chain, improves resource utilization, reduces orchestration costs, can efficiently meet long-term needs in large-scale networks, and protects node data privacy.
Smart Images
Figure CN116192643B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a NFV service function chain dynamic orchestration method based on federated attention reinforcement learning, belonging to the field of service function chain. Background Art
[0002] With the emergence of new network technologies such as cloud computing, software-defined networks, and network function virtualization, future network operation and maintenance will continue to move towards virtualization and intelligence. Network function virtualization provides a method for virtualizing service nodes. It uses general-purpose servers to replace dedicated middleware in traditional networks, which can greatly reduce the construction and operation costs of network operators and improve the flexibility and scalability of network management. Since end-to-end network services usually require different service functions, using virtualization technology to build a network service function chain and reasonably allocate and schedule resources has become an important research topic, which has attracted widespread attention from academia and industry.
[0003] In order to implement the required network services, VNFs should be orchestrated into a suitable virtualized infrastructure in the form of software-based service function chains (SFCs). In the past few years, many SFC orchestration methods have been proposed with different goals, including reducing communication overhead, maximizing network utility, and accelerating performance. Generally speaking, these solutions related to SFC orchestration can be divided into static orchestration and dynamic orchestration.
[0004] Static orchestration instantiates SFC in a given physical device and allocates resources such as storage, computing, and bandwidth. All these instantiated VNFs and SFCs will allocate resources quantitatively throughout the operation. Therefore, the performance of SFC with static orchestration is greatly affected by the current and future network status and actions.
[0005] Dynamic orchestration instantiates SFCs in the available network infrastructure and dynamically allocates network resources for their operation. By proactively learning about traffic trends, these solutions accomplish automatic scaling of VNFs and better utilize re-aggregated SFCs to meet long-term demand. However, these dynamic solutions are still heuristic, slow to react, and difficult to train with large state spaces. Summary of the invention
[0006] The purpose of the present invention is to provide a NFV service function chain dynamic orchestration (SFCDO) method based on federated attention reinforcement learning (FARL), the steps are as follows:
[0007] 1) Read the current physical network topology and service function chain request;
[0008] 2) Generate a placement scheme for virtual network function sets on the physical network based on Federated Attention Reinforcement Learning (FARL);
[0009] 3) Determine whether each virtual network function in the virtual network function set has a server that can be used to place the virtual network function. If so, proceed to step 4); otherwise, proceed to step 6);
[0010] 4) Determine whether there is a link in the physical network that meets the requirements of the placement plan. If so, form a link mapping plan for the service function chain request and proceed to step 5), otherwise proceed to step 6);
[0011] 5) Accept the service function chain request, deploy the service function chain request according to the placement plan and link mapping plan, update the physical network topology, and return to step 1);
[0012] 6) Refuse the service function chain request and return to step 1).
[0013] According to the above technical solution, the steps of generating a placement scheme of a virtual network function set on a physical network based on federated attention reinforcement learning (FARL) in step 2) are as follows:
[0014] Step 1: The cloud server propagates the global model to each client node;
[0015] Step 2: Each client node updates the global model through local iterative training;
[0016] Step 3: The cloud server aggregates the received models into a new global model;
[0017] Step 4: Output the placement solution of the service function chain request based on the trained global model;
[0018] Step 5: Repeat steps 1 to 4 to dynamically optimize the global model.
[0019] According to the above technical solution, step 1, the cloud server propagates the global model to each client node, and the cloud server transmits the global model Θ(t) to the client c i , according to the network scale, node density and client geographical location, the entire network is divided into m sub-networks. In each sub-network, there is a client communicating with the cloud server. i is the index of the client. There are two time periods in the entire architecture. The first is the global federated training round. In each round, there are k local training epchos. Only in the first and kth epchos, the client and the cloud server communicate. t represents the tth federated training round. It is divided into three stages:
[0020] Phase 1: The cloud server initializes a pre-trained model containing parameters based on the attention network architecture.
[0021] Phase 2: The cloud server model is sent to each local client;
[0022] Phase 3: As the local state changes, the model is used to learn a strategy for deploying virtual network functions.
[0023] According to the above technical solution, in step 2, each client node updates the global model through local iterative training, and the client c i Based on the iterative training of the attention neural network and the DQN algorithm, each client iteratively trains its own model in each round based on its own local data (such as CPU, bandwidth, memory, etc. in the network). It is worth noting that the client c i Only the updated model is passed to the cloud server.
[0024] According to the above technical solution, the cloud server in step 3 aggregates the received models into a new global model, and the cloud server collects the model parameters from m clients. By calculation Update the global model parameters, where δ i Indicates the local client c i The weights participating in global model training, where ∑ i δ i =1.
[0025] According to the above technical solution, virtual network function allocation is performed in step 4, and the local client deploys the virtual network function based on the existing global model and calculates the current network overhead and deployment error rate. Steps 1 to 4 are repeated until the global model converges.
[0026] According to the above technical solution, in the attention neural network, each node log is input into its own subgraph, which embeds its log data into a single latent state vector. All subgraphs of a given node type share the same set of parameters, so an increase in the number of nodes does not lead to an increase in the total number of network parameters. The potential state of each node is then stacked and input into the global attention subgraph. This subgraph allows the network to learn which features of neighboring nodes are related to the action value function on a given node. The contextualized state vectors of these nodes are then passed to the feedforward output subgraph. Figure 1 In this way, the network parameters of the output subgraph are shared among nodes of the same type.
[0027] The global self-attention subgraph alternates a multi-head dot-product attention mechanism with an affine layer. This structure makes the output corresponding to node i include relevant information from any other node in the network. Including this learned global context in the latent representation of each node allows the output space to be decomposed into a single subgraph for each node.
[0028] According to the above technical solution, the DQN algorithm, for client c i For local training, we define an action value function Q(S t ,A t ), where S t Indicates the local training state, A t Represents local training behavior. At the same time, γ represents Q(S t ,A t ) Discount factor for subsequent rewards. Optimal action value function Q * (S t ,A t )satisfy:
[0029] The algorithm consists of two networks, the prediction network and the target network. The prediction network is used to evaluate the value function of the current state-action pair. The target function is used to generate the target value. The algorithm updates the parameters in the network according to the update formula of the loss function. After every k rounds of iteration, the parameters in the prediction network are copied to the parameters in the target network. By introducing the target function, the target Q value remains unchanged for a period of time, and the correlation between the predicted Q value and the target Q value is reduced to a certain extent, which reduces the possibility of loss value oscillation divergence during training, thereby improving the stability of the algorithm.
[0030] Both the prediction network and the target network are used to train Q(S t ,A t ; θ), the parameter θ is built-in and can be used to gradually approximate the optimal action-value function Q * (S t ,A t ).
[0031] According to the above technical solution, the local client c i The weights involved in global model training. In order to obtain an optimized global model, two weight calculation methods are set: one based on deployment error rate and the other based on reward. The specific steps are as follows:
[0032] Step 1: Based on the deployment error rate strategy, the deployment accuracy and error rate of each client service function chain are considered separately, δ i It can be calculated as follows: where χ i Represents client c i Deployment error rate
[0033] Step 2: Reward-based strategy, consider client c i The reward χ for the deployment state of the behavior and service function chain i , δi It can be calculated as follows:
[0034] According to the above technical solution, the reward x of the deployment status of the service function chain i , can be calculated as follows: -[σ1c set (t)+σ2c op (t)+σ3c com (t)], where c op (t), c com (t), c set (t) represent the overhead of VNF operation, traffic transmission and VNF local deployment respectively. op (t), c com (t), c set (t), the calculation methods are as follows:
[0035] 1) in represents VNFsf1,f2,…f in the physical network at time t n The number of Indicates f per unit time i operating overhead.
[0036] 2) where ρ s uv (t) represents the traffic of service function chain SFC s passing through link uv at time t.
[0037] 3) in Represents the deployment of VNFs in the network i expenses.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] The present invention provides a method for dynamic orchestration (SFCDO) of NFV service function chains based on federated attention reinforcement learning (FARL). It has three significant features, including: using a federated learning framework to train a global model, protecting the data privacy of each node, and realizing the dynamic orchestration of the service function chain; SFCDO allows the local client and the cloud server to update parameters in the first and last epochs of each round, reducing the communication cost of distributed clients for model optimization; SFCDO introduces an efficient deep reinforcement learning (DRL) method to flexibly map VNFs to the network. In addition, a new neural network architecture based on an attention mechanism is proposed, so that the increase in network nodes will not increase the number of model parameters. The present invention uses federated reinforcement learning and integrates the attention mechanism to realize the dynamic orchestration of NFV service function chains. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is an overall architecture diagram of a NFV service function chain dynamic orchestration method based on federated attention reinforcement learning of the present invention;
[0041] Figure 2 It is a flow chart of a federated attention reinforcement learning (FARL) of a NFV service function chain dynamic orchestration method based on federated attention reinforcement learning of the present invention; DETAILED DESCRIPTION
[0042] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.
[0043] See also Figure 1 The present invention provides a technical solution: a method for dynamic orchestration of NFV service function chains based on federated attention reinforcement learning, comprising the following steps:
[0044] 1. Read the current physical network information and service function chain request (SFC). The virtual network function set requested by the service function chain is denoted as V={VNF1, VNF2, VNF3, ..., VNFn}. n is the number of virtual network functions requested by the service function chain.
[0045] 2See also Figure 2 , based on federated attention reinforcement learning (FARL), a placement scheme for virtual network function sets on the physical network is generated. The steps for generating a placement scheme for virtual network function sets on the physical network are as follows:
[0046] 2.1 Initialization, initialize the global pre-trained model Θ(t) of the cloud server, and initialize the local model Initialize the weight δ of each client i ;
[0047] 2.2 The cloud server transmits the global model Θ(t) to the client c i ,According to the network scale, node density and client geographical location, the entire network is divided into m sub-networks. In each sub-network, there is a client communicating with the cloud server. i is the index of the client. There are two time periods in the entire architecture. The first is the global federated training round. In each round, there are k local training epchos. Only in the first and kth epchos, the client and the cloud server communicate. t represents the tth federated training round.
[0048] 2.3 Client c i Based on the iterative training of the attention neural network and the DQN algorithm, each client iteratively trains its own model in each round based on its own local data (such as CPU, bandwidth, memory, etc. in the network). It is worth noting that the client c i Only the updated model is passed to the cloud server.
[0049] 2.4 For client c i For local training, we define an action value function Q(S t ,A t ), where S t Indicates the local training state, A t Represents local training behavior. At the same time, γ represents Q(S t ,A t ) Discount factor for subsequent rewards. Optimal action value function Q * (S t ,A t )satisfy: Both the prediction network and the target network are used to train Q(S t ,A t ,θ), the parameter θ is built-in and can be used to gradually approximate the optimal action value function Q * (S t ,A t ).
[0050] 2.5 The cloud server collects model parameters from m clients By calculation Update the global model parameters, where δ i Indicates the local client c i The weights participating in global model training, where ∑ i δ i =1.
[0051] 2.6 The local client deploys virtual network functions based on the existing global model and calculates the current network overhead and deployment error rate.
[0052] 2.7 Strategy based on deployment error rate, which considers the deployment accuracy and error rate of each client service function chain, δ i It can be calculated as follows: where χ i Represents client c i Deployment error rate
[0053] 2.8 Reward-based strategy, considering client c i The reward χ for the deployment state of the behavior and service function chaini ;
[0054] 2.9 The reward for the deployment status of the service function chain i , can be calculated as follows: -[σ1c set (t)+σ2c op (t)+σ3c com (t)], where c op (t), c com (t), c set (t) represent the overhead of VNF operation, traffic transmission and VNF local deployment respectively. op (t), c com (t), c set (t), the calculation methods are as follows:
[0055] 2.9.1 in represents VNFsf1,f2,…f in the physical network at time t n The number of Indicates f per unit time i operating overhead.
[0056] 2.9.2 where ρ s uv (t) represents the traffic of service function chain SFC s passing through link uv at time t.
[0057] 2.9.3 in Represents the deployment of VNFs in the network i expenses.
[0058] 2.10 Update client weights based on network overhead and deployment efficiency
[0059] 2.11 Repeat steps 2.1 to 2.10 until the model converges.
[0060] 3. Determine whether each virtual network function in the virtual network function set has a server that can be used to place the virtual network function. If so, proceed to step 4; otherwise, proceed to step 6.
[0061] 4. Determine whether there is a link in the physical network that meets the requirements of the placement plan. If so, form a link mapping plan for the service function chain request and go to step 5. Otherwise, go to step 6.
[0062] 5. Accept the service function chain request, deploy the service function chain request according to the placement plan and link mapping plan, update the physical network topology, and return to step 1;
[0063] 6. Deny the service function chain request and return to step 1.
[0064] The above only expresses the preferred embodiments of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications, improvements and substitutions can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.
Claims
1. A method for dynamic orchestration of NFV service function chains based on federated attention reinforcement learning, characterized in that: The steps include: Step 1: Read the current physical network topology and service function chain request; Step 2: Generate a placement plan for the virtual network function set on the physical network based on federated attention reinforcement learning; Step 3: Determine whether each virtual network function in the virtual network function set has a server that can be used to place the virtual network function. If so, proceed to step 4; otherwise, proceed to step 6; Step 4: Determine whether there is a link in the physical network that meets the requirements of the placement plan. If so, form a link mapping plan for the service function chain request and proceed to step 5. Otherwise, proceed to step 6. Step 5: Accept the service function chain request, deploy the service function chain request according to the placement plan and link mapping plan, update the physical network topology, and return to step 1; Step 6: Refuse the service function chain request and return to step 1; The steps for generating a placement plan for a virtual network function set on a physical network in step 2 are as follows: Step 2.1: The cloud server propagates the global model to each client node; Step 2.2: Each client node updates the global model through local iterative training; Step 2.3: The cloud server aggregates the received models into a new global model; Step 2.4: Output the placement solution Solution of the service function chain request based on the trained global model; Step 2.5: Repeat steps 2.1 to 2.4 to dynamically optimize the global model; Step 2.1 The cloud server propagates the global model to each client node, and the cloud server transmits the global model Θ(t) to the client c i , according to the network scale, node density and client geographical location, the entire network is divided into m sub-networks, each of which has a client communicating with the cloud server, i is the index of the client; the entire architecture has two time periods, the first is the global federated training round, in each round, contains k local training epchos, only in the first and kth epcho, the client and the cloud server communicate, t represents the tth federated training round; the specific steps are as follows: Step 2.11: The cloud server initializes a pre-trained model containing parameters based on the attention network architecture. Step 2.12: The cloud server model is sent to each local client; Step 2.13: As the local state changes, the model is used to learn a strategy for deploying virtual network functions; In step 2.2, each client node updates the global model through local iterative training. i Based on the iterative training of the attention neural network and DQN algorithm, each client iteratively trains its own model in each round based on its own local data. i Only the updated model is passed to the cloud server; In step 2.3, the cloud server aggregates the received models into a new global model. The cloud server collects the model parameters from m clients. By calculation Update the global model parameters, where δ i Represents the local client c i The weights participating in global model training, where ∑ i δ i =1.
2. According to claim 1, a method for dynamic orchestration of NFV service function chains based on federated attention reinforcement learning is characterized in that: In step 2.4, virtual network functions are allocated. The local client deploys virtual network functions based on the existing global model and calculates the current network overhead and deployment error rate. Steps 2.1 to 2.4 are repeated until the global model converges.
3. According to claim 1, a method for dynamic orchestration of NFV service function chains based on federated attention reinforcement learning is characterized in that: In the attention neural network, each node log is input into its own subgraph, which embeds its log data into a single latent state vector; all subgraphs of a given node type share the same set of parameters, so an increase in the number of nodes does not lead to an increase in the total number of network parameters; The latent states of each node are then stacked and fed into a global attention subgraph; this subgraph allows the network to learn which features of neighboring nodes are relevant to the action-value function at a given node; these node-contextualized state vectors are then passed to a feed-forward output subgraph; as with the input subgraph, the network parameters of the output subgraph are shared between nodes of the same type; The global self-attention subgraph alternates a multi-head dot-product attention mechanism with an affine layer; this structure makes the output corresponding to node i include relevant information from any other node in the network; including this learned global context in the latent representation of each node makes the output space decomposed into a single subgraph for each node.
4. According to claim 1, a method for dynamic orchestration of NFV service function chains based on federated attention reinforcement learning, characterized in that: The DQN algorithm, for client c i Local training, defining an action value function Q(S t ,A t ), where S t Indicates the local training state, A t represents the local training behavior; at the same time, γ represents Q(S t ,A t ) Discount factor for subsequent rewards; optimal action value function Q * (S t ,A t )satisfy: The algorithm consists of two networks, a prediction network and a target network. The prediction network is used to evaluate the value function of the current state-action pair. The target function is used to generate the target value. The algorithm updates the parameters in the network according to the update formula of the loss function. After every k rounds of iteration, the parameters in the prediction network are copied to the parameters in the target network. By introducing the target function, the target Q value remains unchanged for a period of time, and the correlation between the predicted Q value and the target Q value is reduced to a certain extent. Both the prediction network and the target network are used to train Q(S t ,A t ; θ), the parameter θ is built-in and is used to gradually approach the optimal action-value function Q * (S t ,A t ).
5. According to claim 1, a method for dynamic orchestration of NFV service function chains based on federated attention reinforcement learning, characterized in that: The local client c i The weights participating in the global model training. In order to obtain an optimized global model, two weight calculation methods are set: one based on the deployment error rate and the other based on the reward. The details are as follows: The strategy based on deployment error rate considers the deployment accuracy and error rate of each client service function chain, δ i This calculation yields: where χ i Represents client c i Deployment error rate Reward-based strategy, consider client c i The reward χ for the deployment state of the behavior and service function chain i , δ i This calculation yields:
6. According to claim 5, a method for dynamic orchestration of NFV service function chains based on federated attention reinforcement learning is characterized in that: The reward x for the deployment status of the service function chain i , calculated in this way: -[σ1c set (t)+σ2c op (t)+σ3c com (t)], where c op (t), c com (t), c set (t) represents the overhead of VNF operation, traffic transmission and VNF local deployment respectively; c op (t), c com (t), c set (t), and the calculation methods are as follows: 1) in represents VNFsf1,f2,…f in the physical network at time t n The number of Indicates f per unit time i operating costs; 2) where ρ s uv (t) represents the traffic of service function chain SFC s passing through link uv at time t; 3) in Represents the deployment of VNFs in the network i expenses.