Service function chain resource dynamic adjustment method based on deep reinforcement learning

Abstract

The invention discloses a service function chain (SFC) dynamic adjustment method based on deep reinforcement learning, and belongs to the field of network function virtualization (NFV) orchestration, and the SFC dynamic adjustment method based on deep reinforcement learning comprises the steps: 1) constructing a data center-oriented physical network model; 2) modeling an SFC request required by a data center; 3) modeling the dynamic adjustment process of the SFC; step 4) modeling in a Markov decision process (MDP); step 5), building an intelligent SFC dynamic adjustment algorithm (ISFCDAA) model; step 6), training an ISFCDAA model; and the trained ISFCDAA is used to carry out SFC dynamic adjustment. According to the method and the device, a relatively high service acceptance rate and long-term income in a data center network can be finally realized, and an effective solution is provided for resource management of a network function virtualization orchestrator (NFVO) orchestrator.

Description

Technical Field

[0001] The present invention relates to the field of NFV orchestration technology, and in particular to a method for dynamically adjusting service function chain resources based on deep reinforcement learning. Background Art

[0002] In the current era of rapid development of Internet technology, users' network service needs are becoming increasingly diversified and complex. Traditional network architecture relies on a series of dedicated hardware devices to implement network functions, including but not limited to firewalls, network address translators, and load balancers. The network services of these dedicated devices are closely related to their hardware, causing service providers to encounter many challenges in responding to changing service needs. Due to the differences in the degree of standardization of different dedicated devices, the flexibility of network configuration is greatly limited. In addition, the high cost of daily maintenance and management of these devices has further increased the cost burden of network operation. In addition, with the rapid development of 5G / 6G network technology and the significant growth in demand for vertical industry services (such as smart manufacturing, telemedicine, etc.), the original network service model based on dedicated hardware devices faces unprecedented challenges. These emerging network services not only put forward higher requirements on the flexibility and scalability of the network, but also have strict requirements on the real-time response capability and quality of service (QoS) of the network. However, due to their inherent hardware dependence and low configuration flexibility, dedicated hardware devices in traditional network architecture are difficult to quickly adapt to such rapidly changing network service needs, resulting in the inability to efficiently support emerging network applications, greatly increasing the cost of network operation, and also restricting network service innovation.

[0003] In response to the challenges faced in traditional network architecture, the European Telecom Standards Institute (ETSI) has proposed a revolutionary network technology solution - Network Function Virtualization (NFV) technology. The core concept of NFV technology is to use virtualization technology to softwareize network functions that were originally fixed on dedicated hardware, so that these network functions can run on general computing platforms, such as industry-standard servers and cloud infrastructure, in the form of virtual network functions (VNFs). This transformation not only greatly improves the deployment flexibility of network services, but also brings new possibilities for network management and service innovation. Through NFV, network service providers can quickly deploy and expand network services without relying on physical hardware, without having to invest in purchasing and maintaining dedicated hardware equipment for each network service. This approach significantly reduces the capital expenditure (CAPEX) and operating expenses (OPEX) of network operations because it reduces the reliance on expensive dedicated hardware, simplifies the maintenance and management of network equipment, and improves resource utilization.

[0004] Although NFV technology has greatly improved the flexibility and cost efficiency of network services, the significant increase in the complexity of network orchestration has undoubtedly posed unprecedented challenges to network management. At this time, the flexibility and programmability of Software Defined Networking (SDN) technology have become the key to solving this problem. The integration of SDN / NFV technology is not just a superposition of technologies, but provides a new perspective for the deployment and management of network services. With the support of SDN / NFV, network services are no longer static configurations, but can be dynamically connected in a specific order through the Service Function Chain (SFC) according to real-time needs to form a VNF forwarding graph that can efficiently process specific network service requests. However, it is this flexibility and dynamism that exacerbates the complexity of orchestration. The NFV manager will effectively deploy VNF instances on standard general-purpose servers according to SFC requirements and physical device capacity limitations, and arrange the dynamic execution order of VNFs to perform SFC scheduling to reduce the total service time. At the same time, due to the continuous changes in SFC requirements, it is also necessary to realize dynamic adjustment of network service resources. However, in the existing SDN / NFV architecture, how to find the optimal solution between the ever-changing network requirements and physical resource limitations, flexibly and efficiently adjust SFC dynamic resources, reduce operating costs, and ensure service continuity and reliability is still an unresolved problem.

[0005] In existing research, there are three main methods for dynamic adjustment of SFC: (a) Horizontal scaling, such as Mao et al., who performed horizontal scaling by increasing or decreasing the number of VNF instances. This strategy can quickly respond to fluctuations in network requests and provide high elasticity and scalability. However, as the number of VNF instances increases, the complexity of configuration and management will also increase, and QoS is difficult to guarantee. (b) Vertical scaling, such as Savi et al., who performed vertical scaling on multi-core CPUs in order to effectively control capital expenditures. Specifically, as data traffic increases or decreases, the computing power of VNF instances also increases or decreases accordingly. However, if the resources of the underlying physical devices are limited or not flexible enough, vertical scaling may be limited and cannot meet dynamically changing needs. (c) Migration, such as Zhao et al., who proposed a migration algorithm that can integrate deployed VNF instances with new network traffic or new network functions by migrating them, but only for some VNFs in SFC and ignores the distribution of remaining resources. These three strategies have their own characteristics, but there may be some restrictions and limitations when applied separately. In the face of many requests, the limited number of physical nodes may be difficult to meet the demand through horizontal scaling. In addition, the introduction of new instances may lead to more complex routing path selection, making it difficult to achieve the required latency standards. At the same time, adopting this strategy often leads to fragmentation in resource usage, reducing overall resource utilization efficiency. In addition, for large-scale network environments, existing adjustment schemes face the problems of high computational complexity and insufficient real-time response capabilities, and more efficient and intelligent adjustment algorithm strategies are urgently needed. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a method for dynamic adjustment of intelligent service function chain (SFC) resources based on deep reinforcement learning. Different from other adjustment methods that only focus on single-target optimization, the present invention solves the problem by comprehensively considering vertical expansion, creating new VNF instances and VNF migration. The strategy can intelligently select appropriate strategies according to demand changes, optimize resource utilization, ensure efficient and balanced distribution of network resources, and minimize related adjustment overhead while meeting user needs.

[0007] In order to achieve the above technical objectives, the technical solution adopted by the present invention is: a method for dynamically adjusting service function chain resources based on deep reinforcement learning, comprising:

[0008] Step 1): Build a physical network model for the data center;

[0009] Step 2): Model the SFC requests required by the data center;

[0010] Step 3): Modeling the SFC dynamic adjustment process;

[0011] Step 4): Markov decision process (MDP) modeling;

[0012] Step 5): Build an intelligent SFC dynamic adjustment algorithm (ISFCDAA) model;

[0013] Step 6): Train the ISFCDAA model and use the trained ISFCDAA to dynamically adjust the SFC.

[0014] As a possible implementation, further, step 1) specifically includes:

[0015] Represent the underlying physical network as an undirected connected graph G = (N, L), where N represents the set of physical servers that can be instantiated with VNFs, and L represents the set of physical links connecting the servers;

[0016] The CPU computing resources of server u∈N are C u , B u,v Represents the link l connecting nodes u and v uv ∈L, the available bandwidth, τ uv Indicates link l uv ∈L transmission delay;

[0017] Each server node is configured with a certain amount of CPU resources to limit the number of requests for sharing VNF instances.

[0018] As a possible implementation, further, step 2) specifically includes:

[0019] In the network, the set of all types of VNFs is represented by F, the set of service requests is represented by I, and the resources required by the same VNF J in different requests are Different; each request i (i∈I) is composed of multiple ordered VNFs, and the VNF set composed of them is composed of F i express;

[0020] Request i (i∈I) is represented by a four-tuple (s i ,t i ,F i ,D i ), s i represents the source end, t i Indicates the destination, D i Indicates the maximum delay allowed by the request; the jth virtual link requested by i is f i,j (f i,j ∈i), b i Represents the bandwidth resources required by i.

[0021] As a possible implementation, further, in step 3), the SFC adjustment includes satisfying the CPU resources of each server, satisfying the actual physical link bandwidth resources, each VNF in the SFC is served by a single server, and satisfying the request latency; the SFC adjustment cost includes the migration cost and the instantiation cost; the SFC adjustment goal includes minimizing the adjustment cost and minimizing the standard deviation of resource distribution.

[0022] As a possible implementation, further, step 4) specifically includes:

[0023] The SFC resource dynamic adjustment problem is modeled as an MDP quintuple (S, A, TP, R, γ), where S is a set of state spaces, A is a set of action spaces, TP is the state transition probability, R is the reward function, and γ is the discount factor;

[0024] The state space is S(t) = {u(t), κ(t)}, where u(t) represents the vector of remaining resources on each physical server at time t, and its expression is κ(t) represents the vector of successful deployment of each type of VNF on the server at time t, that is in Indicates that the VNF type at time t is v x The mapping vector on all servers, i.e. and Indicates that the VNF type at time t is v x Whether to deploy on server n i In the figure, |N| represents the number of servers, |F| represents the number of VNF types, and the state space size is |N|+|N|*|F|;

[0025] The agent selects the best adjustment method (such as vertical expansion, creating a new VNF instance or VNF migration) for the SFC of resource demand changes according to the current state S(t), that is, A(t) = {a1(t),...,a |I| (t)}, where a i (t) represents the dynamic adjustment method selected by i at time t;

[0026] SFC adjustment is successful, the reward is r(t) = -C total (t)-RD(t), where C total (t) is the adjusted total cost, RD(t) is the standard deviation of resource distribution after adjustment; SFC adjustment fails, r(t) = -σ, where σ is much larger than the total cost and resource standard deviation.

[0027] As a possible implementation, further, in step 5), the ISFCDAA model includes a main network and a target network, and the main network and the target network have the same architecture but different parameters;

[0028] Use Pytorch to build a 3-layer fully connected network. The input layer has the same state dimension. The hidden layer contains 128 neurons. All neurons use the linear rectification function ReLU as the activation function.

[0029] The ε-greedy strategy is adopted, and the initial exploration rate is 0.9. As ε(t+1)=max(ε min ,ε(t)-decay rate ) decreased; the memory pool size was set to 600; the Adam optimizer was used, the learning rate was set to 0.015, and the batch size was set to 32; in addition, in order to improve stability and reduce target value deviation, the update cycle was set to 200.

[0030] As a possible implementation, further, in step 6), training the ISFCDAA model specifically includes:

[0031] First, initialize the network parameters, then perform action a by interacting with the environment S(t) t And observe the reward r and next state S of the environment feedback t+1 ; will (S t ,a t ,S t+1 ,r) is called experience, and is stored in the experience playback buffer; (S t ,a t ,S t+1 ,r), we can get Q(S t ,a t ; θ), where θ represents the parameters of the main network;

[0032] S t+1 When input into the main network, the action a corresponding to the largest Q value will be selected t+1 , and S t+1 Input to the target network and get a t+1 The corresponding Q(S t+1 ,a t+1 ; θ'), where θ' represents the parameters of the target network;

[0033] Thus As the actual value of DDQN, Q(S t ,a t ; θ) is used as the predicted value of the network for error back propagation;

[0034] The parameter update formula of the main network is: After several updates, the parameters of the main network are updated to the target network; the above steps are repeated until the predetermined number of training rounds is completed.

[0035] As a possible implementation, further, in step 6), the trained ISFCDAA is used to perform SFC dynamic adjustment, which specifically includes:

[0036] Save the trained network model, input the SFC with resource changes into the algorithm, and obtain the SFC adjustment method π={π1,π2,...}, including adjusting the resources of the VNF on the current node on the SFC, creating an instance of the VNF on the SFC on a new node, and migrating the VNF;

[0037] When VNF on SFC adjusts resources on the current node, if VNF's demand Less than the remaining resources C on the current node u .avi, then adjust the resources on the current node;

[0038] When the action selected is to create an instance of a VNF on a new node on the SFC, first search for candidate nodes that meet resource requirements, then sort the nodes in descending order according to the remaining resources, and then screen the candidate nodes to select the nodes that meet the resource and latency requirements;

[0039] When the action selected is to migrate the VNF, first search for nodes with the same instance, then sort the nodes in descending order according to the remaining resources, and select the node that meets the resource and latency requirements.

[0040] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0041] The present invention provides a dynamic adjustment method of SFC resources based on deep reinforcement learning, constructs the network SFC adjustment process as an MDP model, and thus provides an applicable problem framework for the DRL algorithm. Through the training of the neural network, the ISFCDAA algorithm comprehensively considers vertical expansion, creation of new VNF instances and VNF migration strategies, intelligently selects appropriate strategies according to demand changes, optimizes resource utilization, ensures efficient and balanced distribution of network resources, and minimizes related adjustment overhead while meeting user needs, and maintains high network acceptance rate and long-term benefits, providing an effective solution for resource management of NFV orchestrators. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a simplified flow chart of the present invention;

[0044] Figure 2 This is a schematic diagram of the ISFCDAA training process of the present invention;

[0045] Figure 3 A comparison diagram of the effects of the adjustment method proposed by the present invention and the adjustment methods of other algorithms in terms of resource distribution;

[0046] Figure 4 A comparison chart of the effects of the adjustment method proposed by the present invention and the adjustment methods of other algorithms in terms of service acceptance rate;

[0047] Figure 5 A comparison chart of the effects of the adjustment method proposed by the present invention and the adjustment methods of other algorithms in terms of adjusting the total cost;

[0048] Figure 6 This is a comparison chart of the effects of the adjustment method proposed in the present invention and the adjustment methods of other algorithms in terms of long-term benefits. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0050] Refer to the attached Figure 1 As shown, this embodiment provides a method for dynamically adjusting service function chain resources based on deep reinforcement learning, including:

[0051] Step 1) Build a physical network model for the data center; specifically include:

[0052] The underlying physical network is represented as an undirected connected graph G = (N, L), where N is the set of physical servers that can be instantiated with VNFs, and L is the set of physical links connecting the servers. The CPU computing resources of server u∈N are C u , B u,v Represents the link l connecting nodes u and v uv ∈L available bandwidth resources, τ uv Indicates link l uv ∈ L. Each server node has a certain amount of CPU resources, which limits the number of requests for sharing VNF instances.

[0053] Step 2) Modeling the SFC request required by the data center; specifically including:

[0054] In the network, the set of all types of VNFs is represented by F, the set of service requests is represented by I, and the resources required by the same VNF j (where j is used to distinguish the functions represented by the VNF) in different requests are Different; each request i (i∈I) is composed of multiple ordered VNFs, and the VNF set composed of them is composed of F i Representation; request i (i∈I) is represented by a four-tuple (s i ,t i ,F i ,D i ), s i represents the source end, t i Indicates the destination, D i Indicates the maximum delay allowed by the request; the jth virtual link requested by i is f i,j (f i,j ∈i), b i Represents the bandwidth resources required by i.

[0055] Step 3) Modeling the SFC dynamic adjustment process;

[0056] Among them, SFC adjustment includes satisfying the CPU resources of each server, satisfying the actual physical link bandwidth resources, each VNF in SFC is served by a single server, and satisfying the request latency; SFC adjustment cost includes migration cost and instantiation cost; SFC adjustment goals include minimizing adjustment cost and minimizing resource distribution standard deviation.

[0057] Step 4) Markov Decision Process (MDP) modeling;

[0058] The SFC resource dynamic adjustment problem is modeled as an MDP quintuple (S, A, TP, R, γ), where S is a set of state spaces, A is a set of action spaces, TP is the state transition probability, R is the reward function, and γ is the discount factor;

[0059] Based on the above, the state space is S(t) = {u(t), κ(t)}, where u(t) represents the vector of remaining resources on each physical server at time t, and its expression is κ(t) represents the vector of successful deployment of each type of VNF on the server at time t, that is in Indicates that the VNF type at time t is v x The mapping vector on all servers, i.e. and Indicates that the VNF type at time t is v x Whether to deploy on server n iIn the figure, |N| represents the number of servers, |F| represents the number of VNF types, and the state space size is |N|+|N|*|F|;

[0060] Based on the above, the agent selects the best adjustment method (such as vertical expansion, creating a new VNF instance or VNF migration) for the SFC of resource demand changes according to the current state S(t), that is, A(t) = {a1(t),...,a |I| (t)}, where a i (t) represents the dynamic adjustment method selected by i at time t;

[0061] Based on the above, SFC adjustment is successful, and the immediate reward is r(t) = -C total (t)-RD(t), where C total (t) is the adjusted total cost, RD(t) is the standard deviation of resource distribution after adjustment; SFC adjustment fails, r(t) = -σ, where σ is much larger than the total cost and resource standard deviation.

[0062] Among them, the above TP state transition probability is unknown, because the transition to the next state depends not only on the selected action, but also on external factors, which increase the complexity of the network environment and the difficulty of prediction, making it difficult to directly obtain the state transition probability. Compared with the traditional method that relies on the accurate calculation of TP value, the present invention stores and updates experience through the experience playback mechanism, and can make effective decision adjustments by optimizing the neural network parameters even when the TP cannot be clearly determined.

[0063] Step 5) Building an Intelligent Service Function Chains Dynamic Adaptation Algorithm (ISFCDAA) model;

[0064] The ISFCDAA model includes a main network and a target network, which have the same architecture but different parameters. A three-layer fully connected network is established using Pytorch. The input layer has the same state dimension, and the hidden layer contains 128 neurons. All neurons are activated by the Rectified Linear Unit (ReLU).

[0065] Using the ε-greedy strategy, the initial exploration rate is 0.9, and as ε(t+1)=max(ε min ,ε(t)-decay rate) decreases; in order to improve the sample utilization efficiency of the learning process and reduce the correlation between consecutive samples, the memory pool size is set to 600; and the Adam optimizer is used, the learning rate is set to 0.015, and the batch size is set to 32; in addition, in order to improve stability and reduce target value deviation, the update cycle is set to 200.

[0066] Step 6) Training the ISFCDAA model; specifically including:

[0067] Refer to the attached Figure 2 As shown, first initialize the network parameters; then perform action a by interacting with the environment S(t). t And observe the reward r and next state S of the environment feedback t+1 ; will (S t ,a t ,S t+1 ,r) is called experience, and is stored in the experience playback buffer; (S t ,a t ,S t+1 ,r), we can get Q(S t ,a t ; θ), where θ represents the parameters of the main network; S t+1 When input into the main network, the action a corresponding to the largest Q value will be selected t+1 , and S t+1 Input to the target network and get a t+1 The corresponding Q(S t+1 ,a t+1 ; θ'), where θ' represents the parameters of the target network;

[0068] Thus As the actual value of DDQN ( represents the true Q value of the target network), and Q(S t ,a t ; θ) is used as the predicted value of the network for error back propagation; the parameter update formula of the main network is: After several updates, the parameters of the main network are updated to the target network; the above steps are repeated until the predetermined number of training rounds is completed.

[0069] Save the trained network model, input the SFC with resource changes into the algorithm, and obtain the SFC adjustment method π={π1,π2,...}, including resource adjustment of the VNF on the current node on the SFC, creation of instances of the VNF on the SFC on the new node, and migration of the VNF.

[0070] Based on the above, when the VNF on the SFC adjusts resources on the current node, if the VNF's demand Less than the remaining resources C on the current node u .avi, then adjust the resources on the current node;

[0071] When the action selected is to create an instance of a VNF on a new node on the SFC, first search for candidate nodes that meet resource requirements, then sort the nodes in descending order according to the remaining resources, and then screen the candidate nodes to select the nodes that meet the resource and latency requirements;

[0072] When the action selected is to migrate the VNF, first search for nodes with the same instance, then sort the nodes in descending order according to the remaining resources, and select the node that meets the resource and latency requirements.

[0073] To test the superiority of the adjustment method proposed in this embodiment, the adjustment method provided in the present invention is compared with the adjustment method of DQN and the traditional heuristic REAP in the open source Nobel-Germany network topology. The three algorithms dynamically adjust resources for different numbers of SFCs in a single time slot, and the resource distribution results are shown in the figure. Figure 3 As shown, the average resource distribution standard deviation of the ISFCDAA proposed in the present invention is 2.41% lower than that of DQN and 9.90% lower than that of REAP, indicating that the ISFCDAA proposed in the present invention can achieve the best resource distribution in the network after adjusting resources under different quantities.

[0074] The three algorithms are compared in terms of SFC acceptance rate under 35 time slots, where each time slot has 20 SFCs generating resource changes. The service acceptance rate results are as follows: Figure 4 As shown in Figure 2, the average acceptance rate of ISFCDAA is 15.11% higher than that of DQN and 39.57% higher than that of REAP. Overall, the ISFCDAA algorithm can better adapt to SFC resource changes and achieve a higher acceptance rate through resource adjustment. The total cost of its adjustment is shown in Figure 2. Figure 5 As shown in the figure, the total cost of the algorithm of the present invention is in the middle before the first 5 time slots, which is 16.92% lower than that of DQN and 11.09% higher than that of REAP. After the fifth time slot, the total cost decreases, and although the cost is higher than that of REAP, it is also maintained at a low level, which is 7.99% lower than that of DQN on average; its long-term benefit results are shown in the figure below. Figure 6 As shown, the average long-term return of ISFCDAA is 21.47% higher than that of DQN and 42.92% higher than that of REAP, which proves the flexibility and superiority of the ISFCDAA of the present invention in an environment with dynamic resource changes.

[0075] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for dynamically adjusting service function chain resources based on deep reinforcement learning, characterized in that: include: Step 1): Build a physical network model for the data center; Step 2): Model the SFC requests required by the data center; Step 3): Modeling the SFC dynamic adjustment process; Step 4): Markov decision process MDP modeling; Step 5): Build the intelligent SFC dynamic adjustment algorithm ISFCDAA model; Step 6): Train the ISFCDAA model and use the trained ISFCDAA to dynamically adjust the SFC.

2. The method for dynamically adjusting service function chain resources based on deep reinforcement learning according to claim 1 is characterized in that: Step 1) specifically includes: Represent the underlying physical network as an undirected connected graph G = (N, L), where N represents the set of physical servers that can be instantiated with VNFs, and L represents the set of physical links connecting the servers; The CPU computing resources of server u∈N are C u , B u,v Represents the link l connecting nodes u and v uv ∈L, the available bandwidth, τ uv Indicates link l uv ∈L transmission delay; Each server node is configured with a certain amount of CPU resources to limit the number of requests for sharing VNF instances.

3. The method for dynamically adjusting service function chain resources based on deep reinforcement learning according to claim 2 is characterized in that: Step 2) specifically includes: In the network, the set of all types of VNFs is represented by F, the set of service requests is represented by I, and the resources required by the same VNF j in different requests are Different; each request i (i∈I) is composed of multiple ordered VNFs, and the VNF set composed of them is composed of F i express; Request i (i∈I) is represented by a four-tuple (s i ,t i ,F i ,D i ), s i represents the source end, t i Indicates the destination, D i represents the maximum delay allowed by the request; the jth virtual link requested by i is f i,j (f i,j ∈i), b i Represents the bandwidth resources required by i.

4. The method for dynamically adjusting service function chain resources based on deep reinforcement learning according to claim 1 is characterized in that: In step 3), the SFC adjustment includes satisfying the CPU resources of each server, satisfying the actual physical link bandwidth resources, satisfying each VNF in the SFC being served by a single server, and satisfying the request latency; the SFC adjustment cost includes the migration cost and the instantiation cost; the SFC adjustment goals include minimizing the adjustment cost and minimizing the standard deviation of resource distribution.

5. The method for dynamically adjusting service function chain resources based on deep reinforcement learning according to claim 3 is characterized in that: Step 4) specifically includes: The SFC resource dynamic adjustment problem is modeled as an MDP quintuple (S, A, TP, R, γ), where S is a set of state spaces, A is a set of action spaces, TP is the state transition probability, R is the reward function, and γ is the discount factor; The state space is S(t) = {u(t), κ(t)}, where u(t) represents the vector of remaining resources on each physical server at time t, and its expression is κ(t) represents the vector of successful deployment of each type of VNF on the server at time t, that is in Indicates that the VNF type at time t is v x The mapping vector on all servers, i.e. and Indicates that the VNF type at time t is v x Whether to deploy on server n i In the figure, |N| represents the number of servers, |F| represents the number of VNF types, and the state space size is |N|+|N|*|F|; The agent selects the best adjustment method for the SFC of resource demand changes according to the current state S(t), where the adjustment methods include vertical expansion, creating a new VNF instance, or VNF migration; that is, A(t) = {a1(t),...,a |I| (t)}, where a i (t) represents the dynamic adjustment method selected by i at time t; SFC adjustment is successful, the reward is r(t) = -C total (t)-RD(t), where C total (t) is the adjusted total cost, RD(t) is the standard deviation of resource distribution after adjustment; SFC adjustment fails, r(t) = -σ, where σ is much larger than the total cost and resource standard deviation.

6. The method for dynamically adjusting service function chain resources based on deep reinforcement learning according to claim 1 is characterized in that: In step 5), the ISFCDAA model includes a main network and a target network, wherein the main network and the target network have the same architecture but different parameters; Use Pytorch to build a 3-layer fully connected network. The input layer has the same state dimension. The hidden layer contains 128 neurons. All neurons use the linear rectification function ReLU as the activation function. The ε-greedy strategy is adopted, and the initial exploration rate is 0.

9. As ε(t+1)=max(ε min ,ε(t)-decay rate ) decreased; the memory pool size was set to 600; the Adam optimizer was used, the learning rate was set to 0.015, the batch size was set to 32; and the update period was set to 200.

7. The method for dynamically adjusting service function chain resources based on deep reinforcement learning according to claim 5 is characterized in that: Step 6) trains the ISFCDAA model, specifically including: First, initialize the network parameters, then perform action a by interacting with the environment S(t) t And observe the reward r and next state S of the environment feedback t+1 ; will (S t ,a t ,S t+1 ,r) is called experience, and is stored in the experience playback buffer; (S t ,a t ,S t+1 ,r), we can get Q(S t ,a t ; θ), where θ represents the parameters of the main network; S t+1 When input into the main network, the action a corresponding to the largest Q value will be selected t+1 , and S t+1 Input to the target network and get a t+1 The corresponding Q(S t+1 ,a t+1 ; θ'), where θ' represents the parameters of the target network; Thus As the actual value of DDQN, Q(S t ,a t ; θ) is used as the predicted value of the network for error back propagation; The parameter update formula of the main network is: After several updates, the parameters of the main network are updated to the target network; the above steps are repeated until the predetermined number of training rounds is completed.

8. The method for dynamically adjusting service function chain resources based on deep reinforcement learning according to claim 7 is characterized in that: In step 6), the trained ISFCDAA is used to dynamically adjust the SFC, which specifically includes: Save the trained network model, input the SFC with resource changes into the algorithm, and obtain the SFC adjustment method π={π1,π2,...}, including adjusting the resources of the VNF on the current node on the SFC, creating an instance of the VNF on the SFC on a new node, and migrating the VNF; When VNF on SFC adjusts resources on the current node, if VNF's demand Less than the remaining resources C on the current node u .avi, then adjust the resources on the current node; When the action selected is to create an instance of a VNF on a new node on the SFC, first search for candidate nodes that meet resource requirements, then sort the nodes in descending order according to the remaining resources, and then screen the candidate nodes to select the nodes that meet the resource and latency requirements; When the action selected is to migrate the VNF, first search for nodes with the same instance, then sort the nodes in descending order according to the remaining resources, and select the node that meets the resource and latency requirements.

Images