A service function chain deployment method and system based on deep reinforcement learning algorithm

Abstract

The present invention provides a service function chain deployment method based on a deep reinforcement learning algorithm, comprising the following steps: S1: abstracting an actual network into a mathematical undirected cyclic graph, and expressing a service function chain deployment request as an acyclic directed graph; S2: mathematizing network performance data based on the undirected cyclic graph and the acyclic directed graph; S3: constructing a Markov decision process model based on the mathematized network performance data, comprising a reward set, an action set, and a state set; S4: using an Actor-Critic algorithm to build an Actor network and a Critic network, and extracting hidden state vector features based on a BILSTM neural network, using the features as input to a deep neural network for feature learning, and the deep neural network outputting a policy action set; S5: selecting the policy action with the highest probability of occurrence from the policy action set and applying it to the network environment. The adaptive method of the present invention deploys a service function chain, can effectively adapt to user needs, and significantly improves the deployment success rate of the service function chain in a time-varying network environment.

Description

Technical Field

[0001] The present invention belongs to the field of virtual network technology and specifically relates to a service function chain deployment method and system based on a deep reinforcement learning algorithm. Background Art

[0002] A single network service requires multiple traditional network devices (firewalls, NATs, caches, and load balancers) to provide the service. Service Function Chaining (SFC) abstracts network devices into an ordered collection of virtual network functions (VNFs). By deploying SFCs, VNFs can be flexibly deployed within physical nodes based on network data transmission service requirements, minimizing the impact of intermediate network devices on network data transmission performance.

[0003] In order to improve the success rate of service function chain deployment, scholars have proposed two types of algorithms in recent years: traditional algorithms and adaptive algorithms.

[0004] Traditional algorithms, such as routing, heuristic, or greedy algorithms, suffer from large-scale iteration and computation, resulting in high computational costs and low time efficiency. They also cannot adapt to the QoS constraints of different networks when network conditions change dramatically. To address this issue, some researchers have proposed adaptive algorithms based on reinforcement learning and deep reinforcement learning to optimize the cost and efficiency of service function chain deployment in time-varying environments.

[0005] While deep reinforcement learning algorithms offer a path to adaptive deployment, real-world network environments involve numerous nodes and links, each with its own unique state. Existing reinforcement learning algorithms are unable to effectively extract state space features from a high-dimensional continuous state space. This results in poor training stability for algorithms using network state as prediction input, and significant deviations between training results and optimization objectives.

[0006] Existing deep reinforcement learning algorithms often use a multilayer perceptron (MLP) neural network architecture. This architecture consists of multiple fully connected layers, divided into MLP input, hidden, and output layers. When the number of MLP input and output layers is small, output predictions are relatively accurate. However, when the number of MLP input layers is large, it is necessary to increase the number of hidden layers to build a deeper neural network to improve prediction accuracy. As the number of neural network layers increases, the optimization function is more likely to fall into local optima and deviate from the true global optimum, resulting in worse prediction performance compared to shallower networks.

[0007] Furthermore, because MLP updates model weight parameters based on forward propagation, backward propagation, and gradient update methods, the more weight parameters there are, the greater the time and space overhead of updating the model parameters. MLP utilizes multiple fully connected networks, and the connections within the structure introduce orders of magnitude in weight parameters, which can easily lead to overfitting, vanishing gradients, and exploding gradients, causing the neural network's prediction direction to deviate from the algorithm's model optimization. Therefore, using the MLP structure alone as a deep neural network architecture cannot accurately meet the optimization goals of deploying SFC in deep reinforcement learning algorithms, effectively and accurately deriving the optimal SFC deployment strategy from a high-dimensional continuous state set. Furthermore, MLP cannot model the environment over time series and extract information from that time series. Summary of the Invention

[0008] The purpose of the present invention is to provide a service function chain deployment method and system based on deep reinforcement learning algorithm, which solves the problems of low computational cost and time efficiency and poor algorithm training stability in existing algorithms, resulting in low acceptance rate of service function chain deployment.

[0009] The technical solutions for achieving the purpose of the present invention are as follows:

[0010] On the one hand, the present invention provides a service function chain deployment method based on a deep reinforcement learning algorithm, comprising the following steps:

[0011] Step S110: abstract the actual network into a mathematical undirected cyclic graph, and represent the service function chain deployment request as an acyclic directed graph;

[0012] Step S120: Mathematically converting network performance data according to the undirected cyclic graph and the acyclic directed graph, wherein the network performance data includes latency data, bandwidth data, and network resource data;

[0013] Step S130: Constructing a Markov decision process model based on the mathematized network performance data. The Markov decision process model is a mathematical model that simulates the random strategies and rewards of an intelligent agent in an environment. The mathematical model includes a reward set, an action set, and a state set.

[0014] Step S140: The state set, the action set, and the reward set are constructed using the Actor-Critic algorithm to build an Actor network and a Critic network, and the hidden state vector features of the network environment state space state are extracted based on the BILSTM neural network. The hidden state vector features are used as input to the deep neural network for feature learning, and the deep neural network outputs a set of strategic actions;

[0015] Step S150: Select the policy action with the highest probability of occurrence from the policy action set and apply it to the network environment.

[0016] Based on one aspect, in a possible implementation, the undirected cyclic graph in step S110 is represented as G=(V, E), where G represents a network, V represents a network node, and E represents a link between networks;

[0017] The acyclic directed graph in step S110 is represented by G R =(V R ,E R ), where V R The virtual network function node placed on behalf of the user's service function chain request, E R Represents the set of links between two adjacent virtual network function nodes.

[0018] Based on one aspect, in a possible implementation, the delay data of step S120, the total link transmission delay is less than the transmission threshold of the service function chain delay

[0019] The bandwidth data of step S120 is placed in the service function chain of the adjacent network node u,v∈V. The sum of the requested bandwidth requirements of the virtual network function node cannot exceed the bandwidth B of one of its adjacent network node links. uv ;

[0020] The network resource data of step S120, the sum of the resource requirements of the virtual network function nodes placed on the network node v∈V is less than the sum of the physical resources owned by the network node A v .

[0021] Based on one aspect, in a possible implementation, in step S130, a Markov decision process model is established for the deep reinforcement learning algorithm, and the Markov decision process model is part of the deep reinforcement learning agent.

[0022] Based on one aspect, in a possible implementation, the state set in step S130 is in,

[0023] in, Indicates the remaining resources of each network node, represents the remaining output bandwidth of each network node, and I represents the deployment requirement of the virtual network function node in the SFC deployment request;

[0024] The action set in step S130 is A={0,1,2,3...,|V|}, and the action a∈A is executed each time; the state S at each time t is t ;

[0025] The reward set in step S130 is established based on deep reinforcement learning actions. Deep reinforcement learning actions represent actions of virtual network function nodes. The actions are deployed by the deep reinforcement learning algorithm agent. Each action executes the deployment of a virtual network function node for an SFC deployment request.

[0026] Based on one aspect, in one possible implementation, a reward set is established based on deep reinforcement learning actions, where each successful deployment of a virtual network function node is rewarded and each deployment failure is penalized;

[0027] Deep reinforcement learning algorithms optimize objectives by maximizing long-term rewards.

[0028] Based on one aspect, in a possible implementation, step S140 uses the hidden state feature vector as the input of the MLP neural network, the MLP neural network performs feature learning, and the MLP neural network outputs a policy action set and predicts the probability of the deep reinforcement learning agent taking an action.

[0029] Based on one aspect, in a possible implementation, in step S140, the Actor network is responsible for outputting the strategy action a∈A to interact with the network environment. After receiving the Actor network action, the network environment obtains the reward function R and the state S at the next moment by applying the action to the network environment. t+1 ;

[0030] Status S t+1 And the reward function R is input into the Critic network, and the Critic network outputs the approximate value function

[0031] and state-action value function approximation

[0032] Calculate the advantage function using the TD method:

[0033]

[0034] It is used to evaluate the quality of the Actor network's action at the previous moment. If the action at the previous moment is good, the probability of the action occurring is increased, encouraging the action to continue executing; if the action at the previous moment is not good, the probability of the action occurring is reduced, preventing the action from occurring again.

[0035] The Actor network receives the evaluation of the Critic network and the state set S at this moment. t+1 The policy action is then output through the deep neural network and applied to the network environment again.

[0036] Based on one aspect, in a possible implementation, the deep reinforcement learning agent selects an action with the highest probability of occurrence from the set of strategic actions in step S140 to place a virtual network function node in the network environment;

[0037] Based on the MPLS protocol, an MPLS packet header is allocated to each target flow, and an MPLS forwarding table is set in each network node where the virtual network function node is placed to guide the flow for directional forwarding, so that the flow flows in a directional manner according to the network node where the virtual network function node is placed.

[0038] Another aspect of the present invention provides a service function chain deployment system based on a deep reinforcement learning algorithm, comprising:

[0039] A network modeling unit, which models the service function chain deployment request into an acyclic directed graph and abstracts the actual network into a mathematical undirected cyclic graph, wherein the cyclic undirected graph represents the distribution status of the network environment and the acyclic directed graph represents the service function chain of the network environment;

[0040] a performance optimization unit, wherein the performance optimization unit establishes constraints on delay, bandwidth, and resources based on the undirected cyclic graph and the acyclic directed graph, obtains delay data, bandwidth data, and network resource data, and establishes optimization equations for the delay data, bandwidth data, and network resource data based on an optimization objective;

[0041] A deep reinforcement learning agent unit, comprising: a Markov decision process model established for a deep reinforcement learning algorithm, and an actor-critic algorithm; establishing a state set, an action set, and a reward set based on the deep reinforcement learning algorithm and latency data, bandwidth data, and network resource data; building an actor network and a critic network based on the actor-critic algorithm, and extracting hidden state vector features of the network environment state space state based on a BILSTM neural network. The hidden state vector features are used as input to a deep neural network for feature learning, and the deep neural network outputs a set of strategic actions;

[0042] A deployment unit selects a policy action with the highest probability of occurrence from the policy action set and applies it to the network environment.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This paper, based on the service function chain deployment technology of a deep reinforcement learning algorithm, designs an adaptive method for deploying service function chains. This method can effectively adapt to user needs and significantly improve the success rate of service function chain deployment in time-varying network environments. This method achieves good performance on both wide area networks and wireless (LTE) links, making it applicable to a variety of network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flow chart of a service function chain deployment method based on a deep reinforcement learning algorithm provided by the present invention;

[0046] Figure 2 This is a schematic diagram of the service function chain deployment module structure used in the present invention;

[0047] Figure 3 This is a schematic diagram of the BILSTM neural network structure used in the present invention;

[0048] Figure 4 This is a simulation diagram of the service function chain deployed based on the MPLS protocol in the present invention. DETAILED DESCRIPTION

[0049] In order to further illustrate the technical means and effects adopted by the present invention to achieve the purpose of the predetermined invention, the following is a detailed description of a service function chain deployment method and system based on a deep reinforcement learning algorithm proposed in accordance with the present invention, in combination with the accompanying drawings and specific implementation methods.

[0050] The aforementioned and other technical contents, features, and effects of the present invention are clearly presented in the following detailed description of the specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the intended purpose can be obtained. However, the accompanying drawings are provided for reference and illustration purposes only and are not intended to limit the technical solutions of the present invention.

[0051] In order to solve the problems of low computational cost and time efficiency and poor algorithm training stability in existing algorithms leading to low acceptance rate of service function chain deployment, an embodiment of the present invention provides a service function chain deployment algorithm based on a deep reinforcement learning algorithm.

[0052] See also Figure 1 On the one hand, an embodiment of the present invention provides a service function chain deployment method and system based on a deep reinforcement learning algorithm, comprising the following steps:

[0053] Step S110: abstract the actual network into a mathematical undirected cyclic graph, and represent the service function chain deployment request as an acyclic directed graph.

[0054] The above undirected cyclic graph is represented as G = (V, E), where G represents the network, V represents the network node, and E represents the link between the networks; the above undirected cyclic graph is represented as G R =(V R ,E R ), where V R The virtual network function node placed on behalf of the user's service function chain request, ER Represents the set of links between two adjacent virtual network function nodes.

[0055] The embodiment of the present invention models the network environment as an undirected cyclic graph and the service function chain as an acyclic directed graph. The network environment model is represented as G = (V, E), where G represents the network, V represents the network node, and E represents the link between the networks. The user's data forwarding request, i.e., the service function chain deployment request, is represented as an acyclic directed graph, i.e., G. R =(V R ,E R ), where V R Represents the virtual network function node that the service function chain requests to be placed, E R Represents the set of links between two adjacent VNF nodes.

[0056] Use R to represent the request set of the service function chain R = (r i ,1≤i≤|R|). r i Represents the set of request placement function nodes of the i-th service function chain Indicates the function node required by the i-th service function chain Indicates the latency limit of the deployed service function chain; Indicates the bandwidth limit of the deployed service function chain; Indicates the latency limit of the deployed service function chain.

[0057] To describe the amount of resources of a network node, use v∈V represents the total number of resources of network nodes. Indicates the CPU computing performance of network node v; Measures the storage capacity of network nodes and indicates the memory usage of network nodes; Indicates the remaining processing capacity indicators that network nodes need to meet in the construction of the service function chain. Indicates the amount of resources required to deploy the virtual network function nodes in the service function chain. Indicates the CPU requirements of the virtual network function nodes placed in the service function chain; Indicates the memory requirement of the virtual network function node; Indicates the processing capacity demand of the virtual network function node for the remaining network nodes.

[0058] To describe the deployment request r of the i-th service function chain i Whether the jth virtual network function node on the network is correctly deployed on the network node v, a Boolean variable is defined If the virtual network function node is successfully placed on the network node v, then The value is 1, otherwise it is 0. Defines a Boolean variable Represents the deployment request r of the i-th service function chain i Whether the jth and kth adjacent virtual network function nodes are correctly placed on network nodes u,v, where 1≤j,k≤|r i |, uv∈E. If the adjacent virtual network function nodes are correctly placed and the virtual links are correctly placed in the network links is 1, otherwise it is 0. Considering the link and bandwidth characteristics between network nodes, define L u,v is the delay between network nodes u and v, B u,v is the bandwidth between network nodes u,v, where u,v∈V,uv∈E.

[0059] Step S120: Mathematically convert the network performance data according to the undirected cyclic graph and the acyclic directed graph, wherein the network performance data includes delay data, bandwidth data, and network resource data.

[0060] For the above delay data, the total link transmission delay must be less than the transmission threshold of the service function chain delay. The bandwidth data above, placed in the service function chain of adjacent network nodes u,v∈V, the sum of the requested bandwidth requirements of the virtual network function nodes cannot exceed the bandwidth B of one of its adjacent network node links. uv ; The above network resource data, the sum of the resource requirements of the virtual network function nodes placed on the network node v∈V is less than the sum of the physical resources owned by the network node A v .

[0061] The embodiment of the present invention establishes constraints on latency, bandwidth, and resources based on network conditions, and establishes an optimization equation based on the optimization goal. Specifically, it includes resource constraints. Considering that multiple service instances of a virtual network function node can be deployed on a network node to handle multiple service function chain requests, the sum of the resource requirements of the virtual network function node placed on the network node v∈V is less than the sum of the physical resources owned by the network node A v The total link transmission delay must be less than the transmission threshold of the service function chain delay. The sum of the bandwidth requirements of the virtual network function nodes placed in the service function chain of adjacent network nodes u,v∈V cannot exceed the bandwidth B of one of its adjacent network node links. uv .

[0062] Step S130: Construct a Markov decision process model based on the mathematized network performance data. The Markov decision process model is a mathematical model that simulates the random strategies and rewards of an intelligent agent in an environment. The mathematical model includes a reward set, an action set, and a state set.

[0063] Establish a Markov decision process model for deep reinforcement learning algorithms. The Markov decision process is a mathematical model that simulates the random strategy (policy) and rewards of an intelligent agent in an environment. The Markov decision process model consists of a reward set, an action set, and a state set. The state set is in, in, Indicates the remaining resources of each network node, represents the remaining output bandwidth of each network node, I represents the deployment requirement of the virtual network function node in the SFC deployment request; the action set is A = {0, 1, 2, 3..., |V|}, and the action a∈A is executed each time; the state S at each time t is t The reward set is established based on deep reinforcement learning actions. Deep reinforcement learning actions represent the actions of virtual network function nodes. The actions are deployed by the deep reinforcement learning algorithm agent. Each action executes the deployment of a virtual network function node for an SFC deployment request.

[0064] The embodiment of the present invention establishes a Markov decision process model for a deep reinforcement learning algorithm, that is, based on the deep reinforcement learning process, a state set, an action set, and a reward set are respectively established.

[0065] The state set is represented as in in Indicates the remaining resources of each network node Indicates the remaining output bandwidth of each network node; Indicates the rth i The deployment requirements for the jth virtual network function node of an SFC deployment request are in Indicates the rth i The bandwidth requirements of each service function chain deployment request; Indicates the rth i The latency constraint of each SFC deployment request; Indicates the rth i The resource requirements of the jth virtual network function node of the SFC deployment request; Indicates the rth i The lifetime of a service function chain deployment request.

[0066] Define the action set A = {0, 1, 2, 3..., |V|}, and each action a∈A is executed. At each time t, the state S t , the deep reinforcement learning algorithm agent executes the action of deploying virtual network function nodes, and each action executes the deployment of a virtual network function node for an SFC deployment request. When a=0, it means that the rth i The jth virtual network function node in the SFC deployment request is not successfully deployed on the kth network node. When a=k, it means that the rth i The jth virtual network function node in the SFC deployment request is deployed on the kth network node. Then the action set of the deep reinforcement learning algorithm is:

[0067]

[0068] At the same time, when deploying a virtual network function node in each state, it is necessary to check whether the network resource content of the virtual network function node meets the requirements for deploying the virtual network function; whether the network node deployment delay and link transmission delay consumed by the deployed virtual network function node are less than the delay limit of the SFC deployment request; the transmission bandwidth requirement between adjacent deployed virtual network function nodes cannot exceed the bandwidth B of any adjacent network node link. uv The deployment is successful when the requirements of the virtual network function node for network node resources and the requirements of the SFC for latency and bandwidth resources are met each time. t If the virtual network function node does not meet the node resource deployment requirements, it is necessary to first check whether it meets the SFC's latency and bandwidth resource requirements. If the latency and bandwidth requirements are met, the agent will fall back to the previous state and continue to search for available nodes to deploy the virtual network function node at the next moment until the virtual network function node is successfully deployed. If no network node can be found to place the virtual network function node or the SFC's latency and bandwidth requirements are not met, the placement will be terminated and the SFC's request will be rejected.

[0069] Since SFC is a set of ordered virtual network function nodes, the successful deployment of an SFC is based on the successful deployment of all virtual network function nodes of an SFC. The optimization goal of the deep reinforcement learning algorithm is to maximize the cumulative reward function, that is, the direction of the fastest growth of the reward function represents the best strategy optimization direction of the deep reinforcement learning algorithm at any moment, which reflects the optimization direction formulated by the system model. The reward function is designed so that when the virtual network function node is successfully deployed at each moment, the reward is in For each r i The resources obtained after the jth virtual network function node in the SFC deployment request is successfully deployed. It indicates that F encourages VNF nodes to occupy resources and encourages the placement of VNF nodes, which complies with the optimization goal of minimizing the resource cost of network node maintenance in the SFC system model, so that network nodes can fully receive deployment requests of VNF nodes. Indicates the rth i The network node deployment delay and link transmission delay consumed when deploying the j-th virtual network function node in the SFC deployment request are: The term is used as a penalty coefficient, which meets the optimization goal of minimizing link delay and improving network throughput in the SFC system model. Indicates the rth i The network node bandwidth consumed when deploying the j-th virtual network function node in the SFC deployment request is As a penalty coefficient, it meets the optimization goal of minimizing the bandwidth used by each SFC deployed in the SFC system model and improving the bandwidth utilization of the link.

[0070] In step S140, the state set, action set, and reward set use the Actor-Critic algorithm to build the Actor network and the Critic network, and extract the hidden state vector features of the network environment state space state based on the BILSTM neural network. The hidden state vector features are used as the input of the deep neural network for feature learning, and the deep neural network outputs the strategy action set.

[0071] A reward set is established based on deep reinforcement learning actions. When the virtual network function node is successfully deployed at each moment, it is rewarded, and each deployment failure is punished. The deep reinforcement learning algorithm optimizes the goal by maximizing long-term rewards.

[0072] The hidden state feature vector is used as the input of the MLP neural network, the MLP neural network performs feature learning, and the MLP neural network outputs a set of policy actions and predicts the probability of the deep reinforcement learning agent taking an action.

[0073] The above Actor network is responsible for outputting the strategy action a∈A to interact with the network environment. After receiving the Actor network action, the network environment obtains the reward function R and the state S of the next moment by acting on the action. t+1 ; Status S t+1 And the reward function R is input into the Critic network, and the Critic network outputs the approximate value function and state-action value function approximation Calculate the advantage function using the TD method:

[0074]

[0075] It is used to evaluate the quality of the Actor network's action at the previous moment. If the action at the previous moment is good, the probability of the action will be increased, and the action will be encouraged to continue. If the action at the previous moment is not good, the probability of the action will be reduced, and the action will be avoided again. The Actor network receives the evaluation of the Critic network and the state set S at this moment. t+1 The policy action is then output through the deep neural network and applied to the network environment again.

[0076] Step S150: Select the policy action with the highest probability of occurrence from the policy action set and apply it to the network environment.

[0077] The deep reinforcement learning agent selects the action with the highest probability of occurrence from the set of policy actions in step S140 to place a virtual network function node in the network environment; allocates an MPLS packet header to each target traffic based on the MPLS protocol, and sets an MPLS forwarding table in each network node where the virtual network function node is placed to guide the traffic for directional forwarding, so that the traffic flows in a directional manner according to the network node where the virtual network function node is placed.

[0078] This embodiment of the present invention builds an actor network and a critic network based on the actor-critic algorithm structure. It also extracts hidden state vector features from the state space using the BILSTM neural network. This hidden state feature vector serves as the input to the DNN neural network to predict the probability of an action taken by a deep reinforcement learning agent. The agent in this embodiment selects the action with the highest probability of occurrence and deploys a service function chain within the network based on the SDN architecture and MPLS protocol.

[0079] The embodiment of the present invention also provides a service function chain deployment system based on a deep reinforcement learning algorithm, including a network modeling unit, a performance optimization unit and a deep reinforcement learning agent unit. The network modeling unit models the network environment into an undirected cyclic graph and an acyclic directed graph, wherein the cyclic undirected graph represents the distribution status of the network environment, and the acyclic directed graph represents the service function chain of the network environment; the performance optimization unit establishes constraints on delay, bandwidth and resources according to the undirected cyclic graph and the acyclic directed graph, obtains delay data, bandwidth data and network resource data, and establishes optimization equations for delay data, bandwidth data and network resource data based on the optimization target; the deep reinforcement learning agent unit includes: A Markov decision process model and an Actor-Critic algorithm are established using a deep reinforcement learning algorithm. A state set, action set, and reward set are established based on the deep reinforcement learning algorithm and latency data, bandwidth data, and network resource data. Actor and Critic networks are built based on the Actor-Critic algorithm, and the hidden state vector features of the network environment state space are extracted based on the BILSTM neural network. The hidden state vector features are used as input to the deep neural network for feature learning, and the deep neural network outputs a set of policy actions. The deployment unit selects the policy action with the highest probability of occurrence from the policy action set and applies it to the network environment.

[0080] Figure 2This is a schematic diagram of the service function chain deployment platform of an embodiment of the present invention. First, a Linux system is built. A high-quality virtual switch (Open vSwitch, OVS) that supports multi-layer data forwarding is deployed in the Linux system. The VxLan tunnel mode is selected for the OVS network link. OVS supports OpenFlow, a protocol for remote management in a virtualized network environment. A software-defined network controller RYU controller that supports the OpenFlow protocol is deployed, and a network simulation and orchestration system Mininet is deployed. The virtual switch OVS can be simulated as a virtual network function node with routing functions in an actual network. The network simulation and orchestration system Mininet can simulate OVS and RYU controller into a logical sequence topology of virtual network function nodes that the target network traffic needs to pass through in sequence, namely the service function chain. Mininet builds this topology by running a Python script. The user end can customize the service function chain according to needs using algorithms, ensuring that the target network traffic is transmitted safely, quickly, and stably while minimizing physical resource consumption. At the same time, the Mininet system itself provides a visual topology construction module Miniedit, which provides users with a more intuitive way to orchestrate the topology. This dynamic orchestration approach also enables dynamic allocation of computing and network resources, enabling the construction and deployment of applications tailored to the needs of diverse user applications while also achieving elastic scalability of service capabilities. Compared to traditional simulations using numerous hardware devices, this approach significantly reduces costs. After building a Linux system and deploying a software-defined networking (SDN) controller, following the principle of separating the forwarding and control planes, the SDN controller must receive SFCs issued by users through the SFC platform via a northbound interface, parse these instructions into flow tables, and issue flow control commands via a southbound interface. Using network function virtualization (NFV), actual device functions are virtualized into virtual network function nodes (VNFs). Furthermore, a SQLite database is required to store VNF registration information for the SFC platform and to provide storage for the SDN controller to store unparsed SFCs issued by the SFC platform via the northbound interface, as well as target network traffic sent from users via the northbound interface. Containers are a form of operating system-level virtualization that encapsulates application dependencies, required libraries, and configurations within a package isolated from other containers within the same operating system. Within a pre-built Linux system, deploying Docker, a currently mainstream container technology, can significantly increase the scalability of network emulation supported by the platform. Containers allow applications to run independently and be easily migrated.The virtualization layer in NFV (using the container management system Docker) manages the physical compute, storage, and network components of virtual network function nodes (VNFs) and controls resource allocation for them. Users send the constructed service function chains to the SDN controller via the northbound interface of the software-defined network. The SDN controller parses the service function chains into flow tables. The SDN controller then controls the logical links composed of VNF nodes, which are virtualized using network virtualization technology to create actual routing function components, through the southbound interface of the SDN controller. This ensures that user traffic flows through the desired logical order of the VNF nodes. This allows for debugging of the overall technical approach and refinement of the details. The wireless communication links between network nodes in the air, space, and ground collaborative network are represented in the simulation platform as virtual network links between VNF nodes. A test platform is needed to combine the physical layer transmission characteristics of network links (such as transmission attenuation, transmission delay, antenna gain, bit error rate, noise interference, and other parameters) with specific link layer access protocols. Using a network channel model, these characteristics are converted into performance metrics (primarily network layer transmission rate, delay, and packet loss rate) for virtual network links between virtual network function nodes for quantitative analysis. The Linux operating system provides a rich set of tools for managing and manipulating packet transmission to implement network traffic control, the most widely used of which is Linux Traffic Control (TC). Using Linux TC, virtual network links between simulated nodes are controlled to ensure simulation accuracy and authenticity.

[0081] Figure 3 This is a schematic diagram of a deep neural network module in an embodiment of the present invention. First, the vector matrix is input into the BILSTM to obtain the hidden state matrix H. The hidden states are assigned weights based on the probability of different nodes appearing at this moment:

[0082] E=softmax(W s2 tanh(W s1 H T ))

[0083] Get the new network node compressed hidden state matrix: M = EH, and get the new 4-dimensional node resource state vector S for each column of the hidden state matrix V ', for the n×6-dimensional SFC deployment request state vector input by the n virtual network function nodes of the SFC deployed at this moment, the 6-dimensional state vector is obtained after the BILSTM neural network Will S V 'After passing through two layers of fully connected networks, the action strategy is output and the deployed action is obtained.

[0084] Figure 4 This is a simulation diagram of deploying a service function chain based on the MPLS protocol in an embodiment of the present invention. The figure shows that before the service function chain is deployed, data packets do not flow in the order of the network nodes where the virtual network function nodes are deployed. However, after the virtual network function nodes are deployed, data packets flow in the order of the network nodes where the virtual network function nodes are deployed.

[0085] This embodiment of the present invention, based on the service function chain deployment technology of a deep reinforcement learning algorithm, designs an adaptive method for deploying service function chains. This method effectively adapts to user needs and significantly improves the success rate of service function chain deployment in time-varying network environments. This embodiment of the present invention achieves good performance on both wide area networks and wireless (LTE) links, making it applicable to a variety of network environments.

[0086] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A service function chain deployment method based on deep reinforcement learning algorithm, characterized in that: The following steps are involved: Step S110: abstract the actual network into a mathematical undirected cyclic graph, and represent the service function chain deployment request as an acyclic directed graph; Step S120: Mathematically converting network performance data according to the undirected cyclic graph and the acyclic directed graph, wherein the network performance data includes latency data, bandwidth data, and network resource data; Step S130: Constructing a Markov decision process model based on the mathematized network performance data. The Markov decision process model is a mathematical model that simulates the random strategies and rewards of an intelligent agent in an environment. The mathematical model includes a reward set, an action set, and a state set. Step S140: The state set, the action set, and the reward set are constructed using the Actor-Critic algorithm to build an Actor network and a Critic network, and the hidden state vector features of the network environment state space state are extracted based on the BILSTM neural network. The hidden state vector features are used as input to the deep neural network for feature learning, and the deep neural network outputs a set of strategic actions; Step S150: Select the policy action with the highest probability of occurrence from the policy action set and apply it to the network environment.

2. A service function chain deployment method based on deep reinforcement learning algorithm according to claim 1, characterized in that: The undirected cyclic graph in step S110 is represented by G=(V, E), where G represents the network, V represents the network nodes, and E represents the links between the networks; The acyclic directed graph in step S110 is represented by G R =(V R ,E R ), where V R The virtual network function node placed on behalf of the user's service function chain request, E R Represents the set of links between two adjacent virtual network function nodes.

3. The service function chain deployment method based on deep reinforcement learning algorithm according to claim 1 is characterized in that: The delay data of step S120, the total link transmission delay must be less than the transmission threshold of the service function chain delay The bandwidth data of step S120 is placed in the service function chain of the adjacent network node u,v∈V. The sum of the requested bandwidth requirements of the virtual network function node cannot exceed the bandwidth B of one of its adjacent network node links. uv ; The network resource data of step S120, the sum of the resource requirements of the virtual network function nodes placed on the network node v∈V is less than the sum of the physical resources owned by the network node A v .

4. The service function chain deployment method based on deep reinforcement learning algorithm according to claim 1, characterized in that: In step S130, a Markov decision process model is established for the deep reinforcement learning algorithm. The Markov decision process model is part of the deep reinforcement learning agent.

5. A service function chain deployment method based on deep reinforcement learning algorithm according to claim 1 or 4, characterized in that: The state set in step S130 is in, in, Indicates the remaining resources of each network node, represents the remaining output bandwidth of each network node, and I represents the deployment requirement of the virtual network function node in the SFC deployment request; The action set in step S130 is A={0,1,2,3...,|V|}, and the action a∈A is executed each time; the state S at each time t is t ; The reward set in step S130 is established based on deep reinforcement learning actions. Deep reinforcement learning actions represent actions of virtual network function nodes. The actions are deployed by the deep reinforcement learning algorithm agent. Each action executes the deployment of a virtual network function node for an SFC deployment request.

6. A service function chain deployment method based on deep reinforcement learning algorithm according to claim 5, characterized in that: A reward set is established based on deep reinforcement learning actions. Each time a virtual network function node is successfully deployed, it is rewarded, and each time a deployment fails, it is penalized. Deep reinforcement learning algorithms optimize objectives by maximizing long-term rewards.

7. The service function chain deployment method based on deep reinforcement learning algorithm according to claim 1 is characterized in that: In step S140, the hidden state feature vector is used as the input of the MLP neural network, the MLP neural network performs feature learning, and the MLP neural network outputs a set of policy actions and predicts the probability of the deep reinforcement learning agent taking an action.

8. A service function chain deployment method based on deep reinforcement learning algorithm according to claim 1 or 7, characterized in that: In step S140, the Actor network is responsible for outputting the strategy action a∈A to interact with the network environment. After receiving the Actor network action, the network environment obtains the reward function R and the state S at the next moment by applying the action to the network environment. t+1 ; Status S t+1 And the reward function R is input into the Critic network, and the Critic network outputs the approximate value function and state-action value function approximation Calculate the advantage function using the TD method: It is used to evaluate the quality of the Actor network's action at the previous moment. If the action at the previous moment is good, the probability of the action occurring is increased, encouraging the action to continue executing; if the action at the previous moment is not good, the probability of the action occurring is reduced, preventing the action from occurring again. The Actor network receives the evaluation of the Critic network and the state set S at this moment. t+1 The policy action is then output through the deep neural network and applied to the network environment again.

9. The service function chain deployment method based on deep reinforcement learning algorithm according to claim 1, characterized in that: The deep reinforcement learning agent selects the action with the highest probability of occurrence from the set of strategic actions in step S140 to place a virtual network function node in the network environment; Based on the MPLS protocol, an MPLS packet header is allocated to each target flow, and an MPLS forwarding table is set in each network node where the virtual network function node is placed to guide the flow for directional forwarding, so that the flow flows in a directional manner according to the network node where the virtual network function node is placed.

10. A service function chain deployment system based on deep reinforcement learning algorithm, characterized in that: include: A network modeling unit, which models the service function chain deployment request into an acyclic directed graph and abstracts the actual network into a mathematical undirected cyclic graph, wherein the cyclic undirected graph represents the distribution status of the network environment and the acyclic directed graph represents the service function chain of the network environment; a performance optimization unit, wherein the performance optimization unit establishes constraints on delay, bandwidth, and resources based on the undirected cyclic graph and the acyclic directed graph, obtains delay data, bandwidth data, and network resource data, and establishes optimization equations for the delay data, bandwidth data, and network resource data based on an optimization objective; A deep reinforcement learning agent unit, comprising: a Markov decision process model established for a deep reinforcement learning algorithm, and an actor-critic algorithm; establishing a state set, an action set, and a reward set based on the deep reinforcement learning algorithm and latency data, bandwidth data, and network resource data; building an actor network and a critic network based on the actor-critic algorithm, and extracting hidden state vector features of the network environment state space state based on a BILSTM neural network. The hidden state vector features are used as input to a deep neural network for feature learning, and the deep neural network outputs a set of strategic actions; A deployment unit selects a policy action with the highest probability of occurrence from the policy action set and applies it to the network environment.

Images