Service function chain reliability deployment method, system and device based on deep learning

Abstract

The present invention discloses a service function chain reliability deployment method, system and device based on deep learning. The method includes: S1, obtaining a main topology shortest path set; S2, obtaining the main topology deployment, and calculating the main topology reliability according to the main topology; S3, initializing a finite state space S, a finite action space A and a state transition probability P according to the main topology, and inputting the initialized S, A and P into a CMAC-Q-Learning model, where S and P are used to store state-action pairs generated during the Q-Learning iteration; S4, calculating a value function Q(s t ,a t ); S5. Determine the algorithm's operating conditions. If the conditions are met, execute S4; if not, execute S6. S6. Output the primary topology deployment and backup path deployment to obtain a reliable service function chain deployment solution πs. This invention can achieve reliable service function chain deployment for deep reinforcement learning.

Description

Technical Field

[0001] The present invention relates to the field of service function chains, and in particular to a service function chain reliability deployment method, system and device based on deep learning. Background Art

[0002] Traditional carrier network functions are implemented in hardware middleboxes, capable of achieving "99.999%" high reliability. With the advancement of Network Function Virtualization (NFV) and Virtualized Network Function (VNF) technologies, carrier network functions are increasingly being replaced by virtualized network functions deployed on x86 servers. In a service function chain deployment architecture based on NFV, server nodes in the underlying physical network contain a large amount of data, such as system operation data and contract data. Failure of the underlying general-purpose equipment (virtual machine failure or network function misconfiguration) can cause the mapped service function chain to fail, potentially leading to network downtime. This makes it difficult to guarantee tenant service quality and meet carrier-grade high reliability requirements. Therefore, ensuring the reliability of the service function chain is a pressing issue for providers of underlying physical infrastructure. Summary of the Invention

[0003] The purpose of the present invention is to provide a service function chain reliability deployment method, system and device based on deep learning, aiming to solve the service function chain reliability deployment method, system and device based on deep learning.

[0004] The present invention provides a service function chain reliability deployment method based on deep learning, comprising:

[0005] S1. Obtain the shortest path set of the main topology;

[0006] S2. Obtain the main topology deployment based on the shortest path set, calculate the main topology reliability based on the main topology, and determine whether a backup VNF selection model is required based on the main topology reliability. If a backup VNF selection model is not required, complete the service function chain reliability deployment. If a backup VNF selection model is required, continue executing the steps;

[0007] S3. Initialize the finite state space S, the finite action space A, and the state transition probability P according to the main topology, and input the initialized S, A, and P into the CMAC-Q-Learning model. S and P are used to store the state-action pairs generated during the Q-Learning iteration process.

[0008] S4. Calculation function Q (st ,a t );

[0009] S5. The value function Q(s t ,a t ) is input into the CMAC neural network, and the value function Q(s t ,a t ) Fit the CMAC neural network, update the weight W and take the learning rate α in the local neighborhood W , judge the algorithm running conditions, if the running conditions are met, execute S4, if not, execute S6;

[0010] S6. Calculate the optimal number of backup VNFs B based on the behavior strategy N , based on the joint backup model, the corresponding backup link B is calculated E , according to the computing resources, link resources, delay and reliability constraints, the backup path deployment is obtained, and the main topology deployment and backup path deployment are output to obtain the reliable service function chain deployment solution π s .

[0011] The present invention also provides a service function chain reliability deployment system based on deep reinforcement learning, comprising:

[0012] The acquisition module is used to obtain the shortest path set of the main topology;

[0013] Deployment module: used to obtain the main topology deployment based on the shortest path set, calculate the main topology reliability based on the main topology, and determine whether a backup VNF selection model is needed based on the main topology reliability. If a backup VNF selection model is not needed, the service function chain reliability deployment is completed. If a backup VNF selection model is needed, the execution continues.

[0014] Input module: used to initialize the finite state space S, finite action space A, and state transition probability P according to the main topology, and input the initialized S, A, and P into the CMAC-Q-Learning model. S and P are used to store the state-action pairs generated during the Q-Learning iteration process;

[0015] Calculation module: used to calculate value functions Q (s t ,a t );

[0016] Judgment module: used to convert the value function Q(s t ,a t ) is input into the CMAC neural network, and the value function Q(s t ,a t) Fit the CMAC neural network, update the weight W and take the learning rate α in the local neighborhood W , judge the algorithm running conditions, if the running conditions are met, execute the calculation module, if not, execute the deployment plan module;

[0017] Deployment plan module: Calculate the optimal number of backup VNFs B based on the behavior strategy N , based on the joint backup model, the corresponding backup link B is calculated E , according to the computing resources, link resources, delay and reliability constraints, the backup path deployment is obtained, and the main topology deployment and backup path deployment are output to obtain the reliable service function chain deployment solution π s .

[0018] An embodiment of the present invention also provides a service function chain reliability deployment system based on deep reinforcement learning, comprising: a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein the computer program implements the steps of the above method when executed by the processor.

[0019] An embodiment of the present invention further provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the above method are implemented.

[0020] By adopting the embodiments of the present invention, reliable deployment of service function chains for deep reinforcement learning can be achieved.

[0021] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it is implemented in accordance with the contents of the specification, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 This is a flow chart of a method for deploying service function chain reliability based on deep learning according to an embodiment of the present invention;

[0024] Figure 2 This is a specific flow chart of the service function chain reliability deployment method based on deep learning according to an embodiment of the present invention;

[0025] Figure 3 This is a flow chart of the CMAC-Q-Learning deep reinforcement learning model of the deep learning-based service function chain reliability deployment method according to an embodiment of the present invention.

[0026] Figure 4 2. It is a schematic diagram of a joint backup model of a service function chain reliability deployment method based on deep reinforcement learning according to an embodiment of the present invention;

[0027] Figure 5 is a schematic diagram of a service function chain reliability deployment system based on deep reinforcement learning according to an embodiment of the present invention;

[0028] Figure 6 2 is a schematic diagram of a service function chain reliability deployment device based on deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] Method Example

[0031] According to an embodiment of the present invention, a flow chart of a service function chain reliability deployment method based on deep learning is provided. Figure 1 Flowchart of the service function chain reliability deployment method based on deep learning according to an embodiment of the present invention. Figure 1 As shown, specifically including:

[0032] S1. Obtain the shortest path set of the main topology;

[0033] S2. Obtain the main topology deployment based on the shortest path set, calculate the main topology reliability based on the main topology, and determine whether a backup VNF selection model is required based on the main topology reliability. If a backup VNF selection model is not required, complete the service function chain reliability deployment. If a backup VNF selection model is required, continue executing the steps;

[0034] S3. Initialize the finite state space S, the finite action space A, and the state transition probability P according to the main topology, and input the initialized S, A, and P into the CMAC-Q-Learning model. S and P are used to store the state-action pairs generated during the Q-Learning iteration process.

[0035] S4. Calculation function Q (s t ,at );

[0036] S5. The value function Q(s t ,a t ) is input into the CMAC neural network, and the value function Q(s t ,a t ) Fit the CMAC neural network, update the weight W and take the learning rate α in the local neighborhood W , judge the algorithm running conditions, if the running conditions are met, execute S4, if not, execute S6;

[0037] S6. Calculate the optimal number of backup VNFs B based on the behavior strategy N , based on the joint backup model, the corresponding backup link B is calculated E , according to the computing resources, link resources, delay and reliability constraints, the backup path deployment is obtained, and the main topology deployment and backup path deployment are output to obtain the reliable service function chain deployment solution π s .

[0038] S1 specifically includes: using K-Dijkstra algorithm to calculate the number of nodes n i to n e The shortest path set P is arranged in descending order according to the network delay, and the set is recorded as E={e1,e2,…,e k}, E is the shortest path set of the main topology.

[0039] S2 specifically includes: selecting the shortest path D from the set E with the goal of minimizing end-to-end delay, mapping all VNFs in the service function chain request t to the shortest path D to form a main topology, and calculating the reliability of the main topology. If the service function chain reliability is greater than or equal to the minimum reliability requirement of the service function chain request, there is no need to back up the VNF and the algorithm ends. Otherwise, jump to S3.

[0040] S4 specifically includes:

[0041] For each action a in the finite action space A, where a is the set of backup VNFs, at the current time t, calculate Q NN (s t ,a t ), Q NN (s t ,a t ) represents the state s at time t in the CMAC-Q-Learning model t Execute action a t The value function of will get the set {s t ,a t , Q NN (st ,a t )} is stored in the dataset, in state s t Execute action a t Arrival state s t+1 , we get the profit function r(s t ,a t ), calculate Q NN (s t+1 ,a t+1 ), Q NN (s t+1 ,a t+1 ) represents the state s at time t+1 in the CMAC-Q-Learning model t +1 Execute action a t+1 The value function determines whether rand < 1-ξ1 at the iteration step step+1, where rand represents a randomly generated parameter in the range [0,1], and ξ1 represents the probability of selecting an action. If so, calculate a*=argmax Q NN (s t+1 ,a t+1 ) and δ=r(s t ,a t )+ Q NN (s t+1 ,a t+1 )-Q NN (s t ,a t ), a* represents the action corresponding to the optimal value function. If not, action a is executed based on the action selection strategy π t , calculate Q(s t ,a t ) = Q NN (s t+1 ,a t+1 )+ δ, and update the dataset.

[0042] S5 specifically includes:

[0043] The obtained value function estimate Q(s t ,a t ) Input the CMAC neural network and fit the CMAC neural network, continuously updating the weight W and the learning rate , judgement|Q NN (s t ,a t )-max Q (s t ,a t )| ≥ &&m≤Max_m, Indicates the iteration stopping threshold. If yes, jump to step 4, otherwise jump to step 6. m is the number of iterations, and Max_m is the maximum number of iterations.

[0044] The specific implementation methods are as follows:

[0045] Step 1: Build a service function chain reliability deployment diagram model. The main mathematical symbols and their meanings are shown in Table 1:

[0046] Table 1 Definition of main parameter symbols

[0047]

[0048] Underlying network: Assume that the underlying physical network can be described as an undirected graph G s =(N s ,E s ,M N ,M L ), where N s Represents the set of physical nodes where network functions can be deployed, E s Indicates the connection to the underlying node n s ∈N s The set of physical links, M N Represents the bottom node n s ∈N s Remaining CPU resources, M L Indicates the underlying physical link e s ∈E s The remaining bandwidth resources on each physical node n s ∈N s , the amount of x types of resources that can be provided on a physical node ={ |i∈[1,x]}, where The amount of resources of resource i, including computing resources, storage resources and bandwidth resources, etc. Given a physical node n s ∈N s A set of resources available in the network, any network function f n ∈F requires a set of resources to be instantiated, Represents the set of network functions required in the service function chain request. At the same time, assuming that each server in the underlying network has a certain reliability, the reliability of each server depends on its own working status, r n Indicates the reliability of each server.

[0049] Service function chain request: Assumption Represents a queue consisting of m SFC requests. Each SFC request can be represented as ,in and Represents the server and the user respectively. Represents the set of network functions in the service function chain request t. represents the maximum tolerable delay of service function chain request t, Represents the minimum reliability requirement of the service function chain request t.

[0050] Step 2: Description of the service function chain reliability deployment method:

[0051] (1) A new concept is introduced to describe the reliability of service function chain deployment, and the following definitions are made:

[0052] Definition 1: Reliability. A service function chain can be decomposed into multiple individual components and their connecting links. The reliability of each component is the relative share of time it operates normally, and its reliability can be obtained by the network management system. Therefore, the reliability of a VNF can be expressed by the mean time between failures (MTBF) and the mean time to repair (MTTF). The reliability of a service function chain can be expressed as the ratio of the mean time between failures to the mean time to repair, where the mean time to repair includes the sum of the mean time to failure and the mean time to repair. Therefore, the reliability of a VNF in a service chain can be expressed as:

[0053] (1)

[0054] Each service function chain typically consists of multiple VNFs, and all components within the chain must function normally. Considering only the failure probability of the VNFs, and ensuring that the failure probabilities of each VNF in the chain are independent, VNFs in the same service function chain cannot be mapped to the same physical node. When there are no backup VNFs, the reliability of the service function chain can be expressed as:

[0055] (2)

[0056] If the current service function chain reliability R t Able to meet the reliability requirements of the request Θ t req , then there is no need to deploy any backup VNF. Otherwise, backup VNFs need to be added to improve the reliability and robustness of the service function chain. If any VNF in the service function chain fails, each VNF node has at least one backup function. Assume F b represents the set of all backup VNFs. The reliability of the service function chain after adding the backup VNF is expressed as:

[0057] (3)

[0058] Definition 2: Reliability Improvement Ratio (RIR). The reliability improvement ratio is defined as the ratio of the service function chain reliability after adding the backup VNF to the reliability before adding the backup VNF. The expression of the reliability improvement ratio is as follows:

[0059] (4)

[0060] (2) A new concept is introduced to describe the resource overhead in the deployment process of reliable service function chains, and the following definitions are made:

[0061] Definition 3 Node Importance Factor (NIF). In the underlying network Gs, assume that d i Represents node n s The degree of NIF=(d1, d2,…, d n ) represents node n s The degree of influence on other adjacent nodes.

[0062] Definition 4: Node Connectivity Factor (NCF). Node connectivity indicates the degree of influence of a failed service node on the remaining nodes. Assume that A(G s ) represents G s Adjacency matrix, NCF=NIF·A(G s ) represents the degree of influence of each service node on network connectivity. Therefore, node n s The connectivity can be expressed as:

[0063] (5)

[0064] Definition 5: Edge Connectivity Factor (ECF). Link connectivity indicates the degree of impact of a failed service node on the remaining network links. Link connectivity can be expressed as:

[0065] (6)

[0066] Definition 6: Saturation Factor (SF). Saturation Factor indicates the degree of impact on other service function chains when a service node fails. Therefore, the saturation factor of resource x can be expressed as:

[0067] (7)

[0068] In formula (7), SF(x) represents the saturation of resource x, y represents the number of times resource x can instantiate VNFs, and PMx represents the resource allocation percentage of resource x.

[0069] Definition 7: Resource Stress Factor (RSF). Based on the above definitions of node connectivity, link connectivity, and saturation, the urgency of resource x can be expressed as:

[0070] (8)

[0071] In formula (8), α and β represent weight factors, α + β = 1. Based on this, the resource overhead can be defined as:

[0072] (9)

[0073] In formula (9), C(n s ) and C(e s ) represent the initial resource consumption of nodes and links, respectively. The higher the resource urgency value, the greater the resource overhead of the reliable service function chain deployment. During the deployment process of the reliable service function chain, the resource overhead is divided into two parts: the primary topology mapping overhead and the backup topology mapping overhead. Therefore, the total resource overhead of the service function chain can be expressed as:

[0074] (10)

[0075] In formula (10), μ1, μ2, μ3 and μ4 represent weight factors, which are used to adjust the impact of each resource overhead in different scenarios.

[0076] (3) Establish a service function chain reliability deployment model based on mixed integer linear programming, including the objective function and related constraints:

[0077] Based on this, a mixed integer programming model for the service chain deployment problem under reliability requirements is established, with business reliability requirements, network resources, and latency as constraints, and high reliability and low resource overhead as the goals:

[0078] variable:

[0079] : Binary variable. Introducing binary variables (f n ∈[1,|F|],k∈[1,|N s |]) represents the network function f n Whether to map to physical node k, 1 means selected, 0 means not selected.

[0080] : Binary variable. Introducing binary variables (i∈[1,|S|],j∈[1,|F|],m∈[1,|E sub |]) indicates whether m is selected as the physical link in the main topology mapping process. 1 represents selection, and 0 represents non-selection.

[0081] : Binary variable. Introducing binary variables (f b ∈[1,|F b |],k∈[1,|N s |]) indicates backup network function f b Whether to map to physical node k, 1 means selected, 0 means not selected.

[0082] : Binary variable. Introducing binary variables (f b ∈[1,|F b |],m'∈[1,|E b |]) indicates whether m' is selected as the physical link in the backup topology mapping process. 1 indicates selection, and 0 indicates non-selection.

[0083] h ij : Binary variable. Introduce binary variable h ij (i∈[1,|S|],j∈[1,|F|]) indicates whether the backup VNF is selected for deployment, 1 represents selection, and 0 represents non-selection.

[0084] Optimization goal:

[0085] (11)

[0086] Resource constraints:

[0087] (12)

[0088] (13)

[0089] Reliability constraints:

[0090] (14)

[0091] Connectivity constraints:

[0092] (15)

[0093] (16)

[0094] Network delay constraints:

[0095] (17)

[0096] (18)

[0097] (19)

[0098] Variable constraints:

[0099] (20)

[0100] illustrate:

[0101] Formula (11) is the objective function of the model, which aims to minimize resource consumption while maximizing reliability. U and γ C Represents the weight factor, which is used to adjust the impact of different scenarios on resource consumption and reliability. For example, when the network has high requirements for resource consumption, γ can be appropriately increased. U The weight of U t and C t All were dimensionless.

[0102] Equations (12) and (13) give the resource constraints of nodes and links. Equation (12) is the node resource constraint condition, where Indicates the total computing resource usage in the main topology. represents the total computing resource usage in the backup topology. Formula (13) is the link resource constraint condition, where Indicates the total bandwidth resource usage in the main topology. Indicates the total bandwidth resource usage in the backup topology.

[0103] Formula (14) is the reliability constraint, which means that there are enough backup VNFs in the network. To meet the reliability requirements of service function chain requests.

[0104] Equations (15) and (16) give the connectivity constraints of the service function chain. Equation (15) constrains the inflow and outflow of nodes other than the volume end to be equal based on the traffic balance standard, and Equation (16) ensures that the paths of the primary topology and the backup topology do not conflict with each other.

[0105] Equation (17) is the delay constraint, which means that the delay of the service function chain should meet the service level agreement requirements. The entire end-to-end delay is composed of two parts: Equation (18) and (19). Equation (18) represents the network delay of the service function chain without backup, that is, h ij =0, >0, formula (19) represents the network delay of the service function chain after backup, that is, h ij =1 , >0.

[0106] Formula (20) is a binary constraint, which represents the network function VNF deployment variable 、 and routing variables 、 relationship.

[0107] Step 3: Use Markov decision process to build a backup VNF selection model.

[0108] The Markov Decision Process (MDP) can solve the problem of continuous strategy design and optimization planning in a dynamic environment, especially in scenarios where the underlying network system or network environment changes have a certain degree of dynamic randomness. It is easy to analyze and has intuitive modeling features, making it suitable for modeling the backup VNF selection problem (Backup VNF Selection Model) in the process of reliable service function chain deployment. Therefore, based on the model, the present invention adopts the MDP modeling framework to model and solve the reliable service function chain deployment problem, which can be defined by a five-tuple, namely {S, A, r, P, J}. S represents a finite state space. The basic events in the state space S are shown in Equation (21). X(t) represents the placement state of the VNF in the main topology at time t. Introducing the binary variable X ij (i∈[1,m], j∈[1,n]) represents the deployment of VNFs in the main topology, X ij =1 means select f j Deploy to nodes , 0 means no choice. A represents a finite action space, each action vector a∈ ={ , ,…, } represents the type of VNF in the backup topology and the amount of resources allocated. In order to ensure the consistency of VNF in the main topology and the backup topology, it is necessary to ensure that F = , F = F, where the set is a subset of the set F. r represents the reward set corresponding to the action space. If each state-action set (s, a) satisfies the constraints of the MILP, then by performing action a in state s, the immediate benefit r(s, a) = γ U U t - γ C C tOtherwise, a corresponding penalty factor r=-1 / ξ is given, where ξ represents an infinitesimal positive real number. P represents the state transition probability, J= Represents the total benefit, and T represents the time it takes for the event to arrive. The goal of MDP is to select the action that maximizes the benefit in the decision-making process.

[0109] (twenty one)

[0110] Step 4: Adopt the service function chain deployment method based on deep reinforcement learning, as follows:

[0111] Figure 2 This is a specific flow chart of the service function chain reliability deployment method based on deep learning according to an embodiment of the present invention:

[0112] This paper proposes a reliable service function chain deployment method based on the CMAC-Q-Learning algorithm. This method consists of two main steps: First, for primary topology mapping, the K-Dijkstra algorithm is used to calculate the k shortest paths between the two ends. These k shortest paths are then sorted in descending order based on network latency, and the VNFs in the primary topology are mapped to the selected paths. Second, for backup topology mapping, the backup VNF selection process is modeled as an MDP model. For each VNF in the Markov chain, the CMAC-Q-Learning algorithm is run to determine whether to provide backup resources for the VNF and the amount of resources allocated after accessing the backup VNF. The backup topology is then mapped to the underlying network using a joint backup model, and a reliable service function chain deployment solution πs is output. The specific algorithm process is shown in Table 2.

[0113] Figure 3 This is a flow chart of the CMAC-Q-Learning deep reinforcement learning model of the deep learning-based service function chain reliability deployment method according to an embodiment of the present invention.

[0114] Table 2 Specific process of CMAC-Q-Learning algorithm

[0115]

[0116] Beneficial effects of the present invention:

[0117] The problem of dynamic deployment of service function chains in reliability-required scenarios is to pre-allocate backup resources before mapping, while meeting basic resource and reliability constraints, to reduce the probability of failure of the underlying infrastructure, minimize resource overhead, and meet reliability requirements. Regarding the dynamic deployment of service function chains in reliability-required scenarios, this invention mainly addresses the following three key points:

[0118] ①Build a service function chain backup model with high reliability and low resource overhead;

[0119] ② Design a backup resource optimization mapping method based on deep reinforcement learning to minimize resource costs and obtain maximum reliability, thereby balancing the contradiction between network performance evaluation and optimization.

[0120] ③ Incorporate the value function approximation mechanism into the closed-loop model to reduce the network dimension.

[0121] The service function chain reliability deployment method proposed in the present invention only requires a short number of learning steps to calculate the backup resource plan during the calculation process, and has good iteration speed and solution accuracy. Compared with mainstream backup methods, the service request processing time and the amount of backup resources of the proposed method are reduced to a certain extent. The proposed method can quickly migrate resources to the backup node, maximize the reduction of backup resource overhead, and ensure the best performance of the service function chain reliability. Therefore, the proposed method has the greatest comprehensive advantages in terms of backup resource overhead, service request processing time, and these three aspects, providing an effective technical means for the service function chain reliability deployment of data center networks.

[0122] System Example

[0123] According to an embodiment of the present invention, a service function chain reliability deployment system based on deep reinforcement learning is provided. Figure 5 Schematic diagram of a service function chain reliability deployment system based on deep reinforcement learning according to an embodiment of the present invention. Figure 5 As shown, specifically including:

[0124] The acquisition module is used to obtain the shortest path set of the main topology;

[0125] Deployment module: used to obtain the main topology deployment, calculate the main topology reliability based on the main topology, and determine whether a backup VNF selection model is needed based on the main topology reliability. If a backup VNF selection model is not needed, the service function chain reliability deployment is completed. If a backup VNF selection model is needed, continue to execute the steps;

[0126] Input module: The input is used to initialize the finite state space S, the finite action space A, and the state transition probability P according to the main topology, and input the initialized S, A, and P into the CMAC-Q-Learning model. S and P are used to store the state-action pairs generated during the Q-Learning iteration process;

[0127] Calculation module: used to calculate value functions Q (s t ,a t );

[0128] Judgment module: used to convert the value function Q(s t ,a t ) is input into the CMAC neural network, and the value function Q(s t ,a t ) Fit the CMAC neural network, update the weight W and take the learning rate in the local neighborhood , judge the algorithm running conditions, if the running conditions are met, execute the calculation module, if not, execute the deployment plan module;

[0129] Deployment plan module: Calculate the optimal number of backup VNFs B based on the behavior strategy N , based on the joint backup model, the corresponding backup link B is calculated E , according to the computing resources, link resources, delay and reliability constraints, the backup path deployment is obtained, and the main topology deployment and backup path deployment are output to obtain the reliable service function chain deployment solution π s .

[0130] The acquisition module is specifically used to: use the K-Dijkstra algorithm to calculate the i to n e The shortest path set P is arranged in descending order according to the network delay, and the set is recorded as E={e1,e2,…,e k}, E is the shortest path set of the main topology;

[0131] The deployment module is specifically used to: select the shortest path D from the set E with the goal of minimizing end-to-end delay, map all VNFs in the service function chain request t to the shortest path D to form a main topology, and calculate the reliability of the main topology. If the service function chain reliability is greater than or equal to the minimum reliability requirement of the service function chain request, there is no need to back up the VNF and the algorithm ends. Otherwise, jump to the input module.

[0132] The calculation module is specifically used for:

[0133] For each action a in the finite action space A, where a is the set of backup VNFs, at the current time t, calculate Q NN (s t ,a t ), Q NN (s t ,a t ) represents the state s at time t in the CMAC-Q-Learning model t Execute action a t The value function of will get the set {s t ,a t , Q NN (s t ,at )} is stored in the dataset, in state s t Execute action a t Arrival state s t+1 , we get the profit function r(s t ,a t ), calculate Q NN (s t+1 ,a t+1 ), Q NN (s t+1 ,a t+1 ) represents the state s at time t+1 in the CMAC-Q-Learning model t +1 Execute action a t+1 The value function determines whether rand < 1-ξ1 at the iteration step step+1, where rand represents a randomly generated parameter in the range [0,1], and ξ1 represents the probability of selecting an action. If so, calculate a*=argmax Q NN (s t+1 ,a t+1 ) and δ=r(s t ,a t )+ Q NN (s t+1 ,a t+1 )-Q NN (s t ,a t ), a* represents the action corresponding to the optimal value function. If not, action a is executed based on the action selection strategy π t , calculate Q(s t ,a t ) = Q NN (s t+1 ,a t+1 )+ δ, and update the dataset;

[0134] The judgment module is specifically used for:

[0135] The obtained value function estimate Q(s t ,a t ) Input the CMAC neural network and fit the CMAC neural network, continuously updating the weight W and the learning rate , judgement|Q NN (s t ,a t )-max Q (s t ,a t )| ≥ &&m≤Max_m, Indicates the iteration stopping threshold. If yes, jump to step 4, otherwise jump to step 6. m is the number of iterations, and Max_m is the maximum number of iterations.

[0136] The embodiment of the present invention is a system embodiment corresponding to the above-mentioned method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0137] Device Example 1

[0138] The embodiment of the present invention provides a schematic diagram of a service function chain reliability deployment device based on deep reinforcement learning, such as Figure 6 As shown, it includes: a memory 60, a processor 62 and a computer program stored in the memory 60 and executable on the processor 62. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0139] Device Example 2

[0140] An embodiment of the present invention provides a computer-readable storage medium on which a program for implementing information transmission is stored. When the program is executed by the processor 62, the steps in the above method embodiment are implemented.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. These modifications or replacements of the technical solutions of the embodiments of the present invention do not cause the essence of the corresponding technical solutions to deviate from the scope of this solution.

Claims

1. A service function chain reliability deployment method based on deep reinforcement learning, characterized in that: include, S1. Obtain the shortest path set of the main topology; S2. Obtain the main topology deployment based on the shortest path set, calculate the main topology reliability based on the main topology, and determine whether a backup VNF selection model is required based on the main topology reliability. If a backup VNF selection model is not required, complete the service function chain reliability deployment. If a backup VNF selection model is required, continue executing the steps; S3. Initialize the finite state space S, the finite action space A, and the state transition probability P according to the main topology, and input the initialized S, A, and P into the CMAC-Q-Learning model. S and A are used to store the state-action pairs generated during the Q-Learning iteration process. S4. Calculation function Q (s t ,a t ); S5. The value function Q(s t ,a t ) is input into the CMAC neural network, and the value function Q(s t ,a t ) Fit the CMAC neural network, update the weight W and take the learning rate in the local neighborhood , judge the algorithm running conditions, if the running conditions are met, execute S4, if not, execute S6; S6. Calculate the optimal number of backup VNFs B based on the behavior strategy N , based on the joint backup model, the corresponding backup link B is calculated E , according to the computing resources, link resources, delay and reliability constraints, the backup path deployment is obtained, and the main topology deployment and backup path deployment are output to obtain the reliable service function chain deployment solution π s .

2. The method according to claim 1, characterized in that The S1 specifically includes: using K-Dijkstra algorithm to calculate the number of nodes n i to n e The shortest path set P is arranged in descending order according to the network delay, and is recorded as the set E={e1,e2,…,e k }, E is the shortest path set of the main topology.

3. The method according to claim 2, characterized in that The S2 specifically includes: selecting the shortest path D from the set E with the goal of minimizing the end-to-end delay, mapping all VNFs in the service function chain request t to the shortest path D to form a main topology, and calculating the reliability of the main topology. If the service function chain reliability is greater than or equal to the minimum reliability requirement of the service function chain request, there is no need to back up the VNF and the algorithm ends. Otherwise, jump to S3.

4. The method according to claim 3, characterized in that The S4 specifically includes: For each action a in the finite action space A, where a is the set of backup VNFs, at the current time t, calculate Q NN (s t ,a t ), Q NN (s t ,a t ) represents the state s at time t in the CMAC-Q-Learning model t Execute action a t The value function of will get the set {s t ,a t , Q NN (s t ,a t )} is stored in the dataset, in state s t Execute action a t Arrival state s t+1 , we get the profit function r(s t ,a t ), calculate Q NN (s t+1 ,a t+1 ), Q NN (s t+1 ,a t+1 ) represents the state s at time t+1 in the CMAC-Q-Learning model t +1 Execute action a t+1 The value function determines whether rand < 1-ξ1 at the iteration step step+1, where rand represents a randomly generated parameter in the range [0,1], and ξ1 represents the probability of selecting an action. If so, calculate a*=argmax Q NN (s t+1 ,a t+1 ) and δ = r(s t ,a t )+ Q NN (s t+1 ,a t+1 )-Q NN (s t ,a t ), a* represents the action corresponding to the optimal value function; if not, action a is executed based on the action selection strategy π t , calculate Q(s t ,a t ) = Q NN (s t+1 ,a t+1 )+ δ, and update the dataset.

5. The method according to claim 4, characterized in that The S5 specifically includes: The obtained value function estimate Q(s t ,a t ) Input the CMAC neural network and fit the CMAC neural network, continuously updating the weight W and the learning rate , judgement|Q NN (s t ,a t )-max Q (s t ,a t )| ≥ &&m≤Max_m, Represents the iteration stopping threshold. If yes, jump to step 4, otherwise jump to step 6. m is the number of iterations, and Max_m is the maximum number of iterations.

6. A service function chain reliability deployment system based on deep reinforcement learning, characterized in that: include, The acquisition module is used to obtain the shortest path set of the main topology; Deployment module: used to obtain the main topology deployment based on the shortest path set, calculate the main topology reliability based on the main topology, and determine whether a backup VNF selection model is needed based on the main topology reliability. If a backup VNF selection model is not needed, the service function chain reliability deployment is completed. If a backup VNF selection model is needed, the execution continues. Input module: used to initialize the finite state space S, finite action space A and state transition probability P according to the main topology, and input the initialized S, A and P into the CMAC-Q-Learning model. S and A are used to store the state-action pairs generated during the Q-Learning iteration process; Calculation module: used to calculate value functions Q (s t ,a t ); Judgment module: used to convert the value function Q(s t ,a t ) is input into the CMAC neural network, and the value function Q(s t ,a t ) Fit the CMAC neural network, update the weight W and take the learning rate in the local neighborhood , judge the algorithm running conditions, if the running conditions are met, execute the calculation module, if not, execute the deployment plan module; Deployment plan module: Calculate the optimal number of backup VNFs B based on the behavior strategy N , based on the joint backup model, the corresponding backup link B is calculated E , according to the computing resources, link resources, delay and reliability constraints, the backup path deployment is obtained, and the main topology deployment and backup path deployment are output to obtain the reliable service function chain deployment solution π s .

7. The system according to claim 6, characterized in that The acquisition module is specifically used to: use the K-Dijkstra algorithm to calculate the number of nodes n i to n e The shortest path set P is arranged in descending order according to the network delay, and is recorded as the set E={e1,e2,…,e k }, E is the shortest path set of the main topology; The deployment module is specifically used to: select the shortest path D from the set E with the goal of minimizing end-to-end delay, map all VNFs in the service function chain request t to the shortest path D to form a main topology, and calculate the reliability of the main topology. If the service function chain reliability is greater than or equal to the minimum reliability requirement of the service function chain request, there is no need to back up the VNF and the algorithm ends. Otherwise, jump to the input module.

8. The system according to claim 7, characterized in that The calculation module is specifically used for: For each action a in the finite action space A, where a is the set of backup VNFs, at the current time t, calculate Q NN (s t ,a t ), Q NN (s t ,a t ) represents the state s at time t in the CMAC-Q-Learning model t Execute action a t The value function of will get the set {s t ,a t , Q NN (s t ,a t )} is stored in the dataset, in state s t Execute action a t Arrival state s t+1 , we get the profit function r(s t ,a t ), calculate Q NN (s t+1 ,a t+1 ), Q NN (s t+1 ,a t+1 ) represents the state s at time t+1 in the CMAC-Q-Learning model t +1 Execute action a t+1 The value function determines whether rand < 1-ξ1 at the iteration step step+1, where rand represents a randomly generated parameter in the range [0,1], and ξ1 represents the probability of selecting an action. If so, calculate a*=argmax Q NN (s t+1 ,a t+1 ) and δ = r(s t ,a t )+ Q NN (s t+1 ,a t+1 )-Q NN (s t ,a t ), a* represents the action corresponding to the optimal value function; if not, action a is executed based on the action selection strategy π t , calculate Q(s t ,a t ) = Q NN (s t+1 ,a t+1 )+ δ, and update the dataset; The judgment module is specifically used for: The obtained value function estimate Q(s t ,a t ) Input the CMAC neural network and fit the CMAC neural network, continuously updating the weight W and the learning rate , judgement|Q NN (s t ,a t )-max Q (s t ,a t )| ≥ &&m≤Max_m, Indicates the iteration stopping threshold. If it is, jump to the calculation module, otherwise jump to the deployment plan module. m is the number of iterations, and Max_m is the maximum number of iterations.

9. A service function chain reliability deployment device based on deep reinforcement learning, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the service function chain reliability deployment method based on deep reinforcement learning as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by the processor, it implements the steps of the service function chain reliability deployment method based on deep reinforcement learning as described in any one of claims 1 to 5.

Images