A Method and System for Deploying Intelligent Service Function Chains Based on Deep Reinforcement Learning

Abstract

This invention provides a method and system for deploying intelligent service function chains (SFCs) based on deep reinforcement learning. The method includes: building a neural network prediction model to predict the servers and links occupied by VNFs in the SFC; training the neural network prediction model; and using the trained neural network prediction model to orchestrate the SFC. This invention addresses the SFC orchestration problem in different scenarios by using deep reinforcement learning to construct a Pareto optimal solution set, achieving load balancing while meeting the latency and other requirements of the scenario.

Description

Technical Field

[0001] This invention relates to the field of service function chain deployment, and more specifically, to an intelligent service function chain deployment method and system based on deep reinforcement learning. Background Technology

[0002] As a core communication technology, 5G supports more application scenarios by employing advanced technologies such as Network Functions Virtualization (NFV) and Software Defined Networking (SDN). Its aim is to enhance network flexibility and scalability, providing a large number of users with higher-speed, lower-latency, and customized network services. Based on the differences in service requirements across different scenarios, 5G categorizes application scenarios into three main types: enhanced mobile bandwidth (eMBB), massive machine-type communications (mMTC), and ultra-reliable low-latency communications (uRLLC). For the same metric, the network quality of service (QoS) requirements of users in these scenarios also differ significantly. For example, vehicle-to-everything (V2X) scenarios require real-time perception of vehicle and road conditions, thus demanding higher latency, less than 1ms; while video conferencing and 4K / 8K video scenarios have relatively more lenient latency requirements, less than 10ms and 100ms respectively.

[0003] In NFV and SDN-enabled 5G networks, functions that previously required specialized hardware support have become Virtualized Network Functions (VNFs) that can run on general-purpose equipment. This significantly reduces the cost and operating expenses of 5G networks, while increasing network flexibility and scalability. Specifically, when a user issues a service request, the service operator assembles the required VNFs into a Service Function Chaining (SFC) and deploys each VNF in the SFC sequentially on a server using intelligent algorithms. During deployment, the resources of network servers and links must meet the requirements of running and transmitting VNFs (such as computing resources and bandwidth). This resource allocation problem is known as the SFC orchestration problem, which has been proven to be NP-hard.

[0004] In SFC orchestration, meeting the latency and load balancing requirements of VNFs is crucial for ensuring optimal performance and user experience. Specifically, since different business needs have varying latency requirements, strategically placing VNFs with different latency levels within a given timeframe is fundamental for operators to provide satisfactory service to users. Furthermore, with the current surge in network requests, effectively balancing the workload across servers and preventing network crashes due to excessive server load is another specific challenge faced by SFC orchestration.

[0005] With the continuous development of intelligent algorithms, significant progress has been made in single-objective optimization algorithms for the SFC orchestration problem. Liu et al. proposed an SFC-DOP algorithm based on Deep Reinforcement Learning (DRL) to deploy user requests, aiming to optimize end-to-end latency, but it neglected the load balancing capabilities of nodes. To improve SFC acceptance rate, Li et al. proposed a VNF mapping scheduling algorithm based on tabu search, but it did not consider the diversity of user requests in the overall network environment. Considering the real-time and random nature of user requests, Fan et al. used graph convolutional neural networks to learn the characteristics of the underlying network environment online, and simultaneously used heuristic algorithms and DRL to optimize the SFC orchestration model, but they did not solve the latency and load balancing problems in the online orchestration process. Hurmat et al. modeled the SFC deployment problem as a Markov decision process and aimed to increase supplier benefits by using multiple DRL agents to handle multiple SFC placements simultaneously. However, these algorithms have significant limitations in solving deployment problems because traditional single-objective optimization algorithms do not consider the diversity of QoS in different scenarios. Therefore, how to meet the QoS requirements of users in different scenarios from multiple perspectives remains a challenge. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and system for deploying intelligent service function chains (SFCs) based on deep reinforcement learning. Unlike other SFC orchestration methods that only focus on the same scenario, this invention addresses SFC orchestration problems in different scenarios by using deep reinforcement learning to construct a Pareto optimal solution set, thereby maximizing load balancing while meeting the latency and other requirements of each scenario.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] The first aspect of this invention provides a method for deploying intelligent service function chains based on deep reinforcement learning, comprising:

[0009] Build a neural network prediction model to predict the servers and links occupied by VNFs in SFC;

[0010] Train the neural network prediction model;

[0011] Use a pre-trained neural network prediction model to orchestrate SFC.

[0012] Based on the above, the neural network prediction model built for predicting the servers and links occupied by VNFs in SFC first predicts the occupied servers and then calculates the recommended paths between the connected servers.

[0013] Based on the above, the model used to predict the occupied servers in the neural network prediction model consists of multiple neural network models with the same structure.

[0014] The neural network model includes an input layer, an encoder, a decoder, and an output layer;

[0015] The input layer is used to process and normalize the computing resources, degree, and bandwidth resources of the servers in the underlying network into a 3×n matrix. f The characteristic matrix of , where n f This refers to the number of servers in the underlying network.

[0016] The encoder consists of multiple Long Short-Term Memory Recurrent Neural Networks (LSTMs). The LSTMs extract the final hidden state c of the underlying network through three gate structures. t t = 1, 2, ..., n f ;

[0017] The decoder consists of multiple LSTMs, and its input is the last hidden state of the encoder. After a series of LSTM operations, the final output is obtained. in, Let be the probability of selecting the t-th server;

[0018] The output layer employs a sampling function to sample the output probability in the decoder to obtain the occupied server.

[0019] Based on the above, the Long Short-Term Memory Recurrent Neural Network (LSTM) extracts the final hidden state c of the underlying network through three gate structures. t The method is as follows:

[0020] The first gate structure is set as a forget gate, which stores the feature x of this node. t and the output h of the previous LSTM t-1 Perform a non-linear mapping and output a vector f. t =σ(θ) f (h f-1 +x t )+b f ), where θ fand b f The weights and offsets of the forget gate structure, f t The value range is between [0, 1], representing different degrees of retention, where 0 means no retention at all and 1 means full retention.

[0021] The second gate structure is set as the input gate, and the value i to be updated is determined first. t =σ(θ) i (h t-1 +x t )+b i Then, candidate vectors for adding states are generated. Last update cell status Where, θ i and b i The weights and offsets of the input gate structure, θ c and b c These are the weights and offsets of the tanh structure;

[0022] The third gate structure is set as an output gate, used to output the final value h. t =σ(θ0(h) t-1 +x t )+b0)tanh(c t The output of ), where θ0 and b0 are the weights and offsets of the output gate structure;

[0023] Based on the above, the recommended path algorithm in the neural network prediction model is as follows:

[0024] The first step is to select the starting server p. i With the endpoint server p i+1 Initially, i is 1;

[0025] Where i = 1, 2, ..., n Z ; For all servers that are predicted to be in use;

[0026] The second step is to set up the starting server p. i Add the point to the known set, set the flag to true to indicate that the shortest path from the point to the source has been determined, and add the points to the other servers p. m Add to an unknown set;

[0027] Third step: Update the starting server p i Other servers p m The consumption of dis(p) i p m );

[0028] Fourth step, in the unknown set, select dis(p) i p m The server with the smallest median value px , will server p x Add to a known set;

[0029] Fifth step, among the remaining servers, calculate dis(p) i p m )>dis(p i p x )+dis(p x p m );

[0030] If this inequality holds, then dis(p) i p m ) = dis(p i p x )+dis(p x p m ), at this time p i With p m The path passes through p x ;

[0031] Loop until p i+1 When a point is added to a known set, the dis(p) value is obtained. i p i+1 That is, the shortest distance;

[0032] Step 6: After obtaining the shortest distance between the two servers, let i = i + 1, and return to step 1 to select a new starting server and a new ending server, until the shortest path between all servers has been calculated.

[0033] Based on the above, assume that all neural network models are based on the objective function F;

[0034] F = a × la + (1-a) × lo + pe;

[0035] Where, La represents the service latency; Lo represents the orchestrated load balancing score; Pe represents the penalty for SFC orchestration failure, which is 0 when SFC orchestration is successful and 10 when it fails; the value range of a is [0, 1], indicating the degree of importance attached to latency and load balancing; when a is 1, it means that only latency is considered and load balancing is not considered, and when a is 0, it means that only load balancing is considered and latency is not considered; n s In the aforementioned neural network models, the difference in the value of 'a' between two adjacent neural network models is 1 / n. s The objective function for each of the aforementioned neural network models is different.

[0036] When training adjacent neural network models, the parameters of the initial neural network model are taken from the parameters of the previously trained neural network model.

[0037] Each of the aforementioned neural network models is trained using the policy gradient algorithm, as follows:

[0038] The first step is to collect past business data from operators and simulate the underlying network resources and SFC request situation in real life based on this real data;

[0039] The second step is to set the number of neural network models to n. S Each neural network model requires a training batch of rounds with a learning rate of lr = 0.001. The neural network parameters are updated every M SFC requests processed.

[0040] The third step is to initialize the parameters θ in the current neural network model. k，y If it is not the first neural network model, then use the parameters of the previously trained neural network model; k = 0, 1, ..., (n s -1);y=1,2,…,batch;

[0041] Set the objective function of the neural network model to F = a × La + (1-a) × Lo + Pe. If it is the first round, a is 1.

[0042] Fourth step: When there is an SFC request, repeat this step repeatedly:

[0043] The underlying network resource information is then input into the neural network prediction model to obtain the placement location and the occupied links;

[0044] The allocation scheme is evaluated using an objective function to obtain the latency La and the load balancing score Lo of the server and link. The load balancing situation is represented by the variance of the resource occupancy ratio of each server and link.

[0045] Next, determine whether the orchestration was successful, that is, whether the computing and bandwidth resources in the underlying network can meet the needs of SFC;

[0046] The decoder predicts the server probability output. j And evaluation results F j Store in the cache. When the number of requests to SFC is equal to M, proceed to step 5; j = 0, 1, ..., M.

[0047] Fifth step, update the parameters θ of the currently trained neural network model. k，y ;

[0048]

[0049] Where lr is the learning rate of the currently trained neural network model;

[0050] Step 6: Clear the cache and return to step 3 to continue training;

[0051] Step 7: When the number of training rounds exceeds the batch size, save the currently trained neural network model; change the parameter a = a⁻¹ / n in the objective function F of the neural network model. s Then return to step three for training, until n s All neural network models have been trained.

[0052] Based on the above, when orchestrating SFCs, the underlying network resources are first collected, then the Pareto front is constructed using the collected information, and finally a greedy selection is performed.

[0053] Based on the above, the method for arranging SFCs is as follows:

[0054] The first step is to collect the underlying network resource information of the current time slice after receiving the SFC request, and process this information into data that is convenient for the neural network prediction model to calculate.

[0055] The second step involves inputting the collected underlying network resource data into the neural network prediction model to construct the Pareto optimal solution set for this problem:

[0056] Simultaneously, the underlying network resource data is input into the n of the neural network prediction model. S In the neural network model, n is obtained s This arrangement method, n s This arrangement method is the Pareto optimal solution set for this problem;

[0057] The third step is to traverse each arrangement scheme in the Pareto optimal solution set and find the scheme that satisfies min(Lo) under the condition la≤L, where L is the user's maximum allowed delay.

[0058] A second aspect of the present invention provides an intelligent service function chain deployment system based on deep reinforcement learning, comprising:

[0059] Memory; and

[0060] A processor coupled to the memory is configured to execute the deep reinforcement learning-based intelligent service function chain deployment method based on instructions stored in the memory.

[0061] A third aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned intelligent service function chain deployment method based on deep reinforcement learning.

[0062] This invention has outstanding substantive features and significant progress compared to the prior art, specifically:

[0063] Unlike other single-scenario methods, this patent can meet the network quality requirements of users in different scenarios while balancing the network load as much as possible, improving the overall network stability and preventing network congestion. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the composition of the neural network model of the present invention.

[0065] Figure 2 This is a schematic diagram of the encoder of the present invention.

[0066] Figure 3 This is the method flow for arranging SFCs in this invention.

[0067] Figure 4 This is a comparison chart showing the effects of the arrangement scheme proposed in this invention and the traditional heuristic arrangement method. Detailed Implementation

[0068] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0069] Example 1

[0070] This embodiment provides a method for deploying intelligent service function chains based on deep reinforcement learning, including:

[0071] Step 1: Build a neural network prediction model to predict the servers and links occupied by VNFs in SFC;

[0072] The neural network prediction model built to predict the servers and links occupied by VNFs in SFC first predicts the occupied servers and then calculates the recommended paths between the connected servers.

[0073] Specifically, the neural network prediction model used to predict the occupied servers consists of multiple neural network models with the same structure.

[0074] like Figure 1 As shown, the neural network model includes an input layer, an encoder, a decoder, and an output layer;

[0075] The input layer is used to process and normalize the computing resources, degree, and bandwidth resources of the servers in the underlying network into a 3×n matrix. f The characteristic matrix of , where n f This represents the number of servers in the underlying network.

[0076] The encoder, such as Figure 2 As shown, it consists of multiple Long Short-Term Memory Recurrent Neural Networks (LSTM). The LSTM extracts the final hidden state c of the underlying network through three gate structures.t t = 1, 2, ..., n f ;

[0077] The Long Short-Term Memory Recurrent Neural Network (LSTM) extracts the final hidden state c of the underlying network through three gate structures. t The method is as follows:

[0078] The first gate structure is set as a forget gate, which stores the feature x of this node. t and the output h of the previous LSTM t-1 Perform a non-linear mapping and output a vector f. t =σ(θ) f (h f-1 +x t )+b f ), where θ f and b f The weights and offsets of the forget gate structure, f t The value range is between [0, 1], representing different degrees of retention, where 0 means no retention at all and 1 means full retention.

[0079] The second gate structure is set as the input gate, and the value i to be updated is determined first. t =σ(θ) i (h t-1 +x t )+b i Then, candidate vectors for adding states are generated. Last update cell status Where, θ i and b i The weights and offsets of the input gate structure, θ c and b c These are the weights and offsets of the tanh structure;

[0080] The third gate structure is set as an output gate, used to output the final value h. t =σ(θ0(h) t-1 +x t )+b0)tanh(c t The output of ), where θ0 and b0 are the weights and offsets of the output gate structure.

[0081] The decoder, like the encoder, consists of multiple LSTMs, and its input is the last hidden state of the encoder. After a series of LSTM operations, the final output is obtained. in, Let n be the probability of selecting the nth server.

[0082] The output layer employs a sampling function to sample the output probability in the decoder to obtain the occupied server.

[0083] When the neural network model predicts all the servers in use Then, Dijkstra's algorithm is used to calculate the occupied links; specifically:

[0084] The first step is to select the starting server p. i With the endpoint server p i+1 Initially, i is 1; i = 1, 2, ..., n Z ;

[0085] The second step is to set up the starting server p. i Add the point to the known set, set the flag to true to indicate that the shortest path from the point to the source has been determined, and add the other servers p. m Add to an unknown set;

[0086] Third step: Update the starting server p i Other servers p m The consumption of dis(p) i p m );

[0087] Fourth step, in the unknown set, select dis(p) i p m The point server p with the smallest median value x , will point server p i Add to a known set;

[0088] Fifth step, among the remaining servers, calculate dis(p) i p m )>dis(p i p x )+dis(p x p m );

[0089] If this inequality holds, then dis(p) i p m ) = dis(p i p x )+dis(p x p m ), at this time p i With p m The path passes through p x ;

[0090] Loop until p i+1 When a point is added to a known set, the dis(p) value is obtained. i p i+1 That is, the shortest distance;

[0091] Step 6: After obtaining the shortest distance between the two servers, let i = i + 1, and return to step 1 to select a new starting server and a new ending server, until the shortest path between all servers has been calculated.

[0092] Step 2: Train the neural network prediction model.

[0093] Step 2 is responsible for training the parameters of all neural network models in the neural network prediction model, specifically as follows:

[0094] Assume that all neural network models are based on the objective function F;

[0095] F = a × la + (1-a) × lo + pe;

[0096] Where, La represents the service latency; Lo represents the orchestrated load balancing score; Pe represents the penalty for SFC orchestration failure, which is 0 when SFC orchestration is successful and 10 when it fails; the value range of a is [0, 1], indicating the degree of importance attached to latency and load balancing; when a is 1, it means that only latency is considered and load balancing is not considered, and when a is 0, it means that only load balancing is considered and latency is not considered; n s In the aforementioned neural network models, the difference in the value of 'a' between two adjacent neural network models is 1 / n. s The objective function for each of the aforementioned neural network models is different.

[0097] When training adjacent neural network models, the parameters of the initial neural network model are taken from those of the previously trained neural network model. This can significantly shorten the training time because the objective functions of adjacent neural network models are similar and their model parameters are not much different.

[0098] Furthermore, each of the aforementioned neural network models is trained using a policy gradient algorithm, as follows:

[0099] The first step is to collect past business data from operators and simulate the underlying network resources and SFC request situation in real life based on this real data;

[0100] The second step is to set the number of neural network models to n. S Each neural network model requires a training batch of rounds with a learning rate of lr = 0.001. The neural network parameters are updated every M SFC requests processed.

[0101] The third step is to initialize the parameters θ in the current neural network model. k，y If it is not the first neural network model, then use the parameters of the previously trained neural network model; k = 0, 1, ..., (n s-1);y=1,2,…,batch;

[0102] Set the objective function of the neural network model to F = a × La + (1-a) × Lo + Pe. If it is the first round, a is 1.

[0103] Fourth step: When there is an SFC request, repeat this step repeatedly:

[0104] The underlying network resource information is then input into the neural network prediction model to obtain the placement location and the occupied links;

[0105] The allocation scheme is evaluated using an objective function to obtain the latency La and the load balancing score Lo of the server and link. The load balancing situation is represented by the variance of the resource occupancy ratio of each server and link.

[0106] Next, determine whether the orchestration was successful, that is, whether the computing and bandwidth resources in the underlying network can meet the needs of SFC;

[0107] The decoder predicts the server probability output. j And evaluation results F j Store in the cache. When the number of requests to SFC is equal to M, proceed to step 5; j = 0, 1, ..., M.

[0108] Fifth step, update the parameters θ of the currently trained neural network model. k，y ;

[0109]

[0110] Where lr is the learning rate of the currently trained neural network model;

[0111] Step 6: Clear the cache and return to step 3 to continue training;

[0112] Step 7: When the number of training rounds exceeds the batch size, save the currently trained neural network model; change the parameter a = a⁻¹ / n in the objective function F of the neural network model. s Then return to step three for training, until n s All neural network models have been trained.

[0113] Step 3: Use the pre-trained neural network prediction model to orchestrate SFC.

[0114] To simultaneously meet users' real-world requirements for service latency and load balancing, when orchestrating SFCs, the underlying network resources are first collected, then a Pareto front is constructed using the collected information, and finally a greedy selection is performed.

[0115] like Figure 3As shown, the specific method for arranging SFCs is as follows:

[0116] The first step is to collect the underlying network resource information of the current time slice after receiving the SFC request, and process this information into data that is convenient for the neural network prediction model to calculate.

[0117] The second step involves inputting the collected underlying network resource data into the neural network prediction model to construct the Pareto optimal solution set for this problem:

[0118] Simultaneously, the underlying network resource data is input into the n of the neural network prediction model. S In the neural network model, n is obtained s This arrangement method, n s This arrangement method is the Pareto optimal solution set for this problem;

[0119] The third step is to traverse each arrangement scheme in the Pareto optimal solution set and find the scheme that satisfies min(Lo) under the condition la≤L, where L is the user's maximum allowed delay.

[0120] To test the superiority of the orchestration scheme proposed in this embodiment, the orchestration scheme and traditional heuristic orchestration methods were processed in an open-source ABI network topology to handle SFC requests arriving according to a Poisson distribution. The results of processing over 2000 time slots are as follows: Figure 4 As shown, the request acceptance rate of this orchestration scheme is around 96%, while the request acceptance rates of the first-fit algorithm and the legacy algorithm are around 85% and 93%, respectively, indicating that the proposed orchestration scheme is superior to traditional solutions.

[0121] Example 2

[0122] This embodiment provides an intelligent service function chain deployment system based on deep reinforcement learning, including:

[0123] Memory; and

[0124] A processor coupled to the memory is configured to execute the deep reinforcement learning-based intelligent service function chain deployment method described in Example 1 based on instructions stored in the memory.

[0125] Example 3

[0126] This embodiment provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the intelligent service function chain deployment method based on deep reinforcement learning described in Embodiment 1.

[0127] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-non-transitory readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0128] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1.A method for intelligent service function chaining deployment based on deep reinforcement learning, characterized in that, The method comprises the following steps: building a neural network prediction model for predicting the server and link occupied by a virtual network function VNF in a service function chain SFC; training the neural network prediction model; setting all neural network models based on an objective function F; F = a x l a + (1 - a) x l o + pe; Wherein, La is the latency of providing services; Lo is the load balancing score after arrangement; Pe is the penalty of service function chain SFC arrangement failure, which is 0 when the service function chain SFC arrangement is successful and 10 when the service function chain SFC arrangement fails; the value range of a is [0, 1], which represents the degree of attention to latency and load balancing; when a is 1, it means only paying attention to latency and not paying attention to load balancing, and when a is 0, it means only paying attention to load balancing and not paying attention to latency; n s The value difference between adjacent two neural network models is 1 / n s ; the objective function of each neural network model is different; when training an adjacent neural network model, the initial parameters of the current neural network model use the parameters of the last trained neural network model adjacent thereto; each neural network model is trained by using a policy gradient algorithm, and the process is as follows: Step 1: Collect the past business data of a 5G service operator, and simulate the underlying network resources and service function chain SFC request conditions in real life according to the business data; Second step, set the number of neural network models as n s Each neural network model needs to be trained for batch rounds, the learning rate lr=0.001, and the neural network parameters are updated every M service function chain (SFC) request. Step 3, initialize the parameters θ in the current neural network model k,y If not the first neural network model, use the parameters of the last trained neural network model adjacent to the current neural network model; k = 0, 1, …, (n s - 1); y = 1, 2, …, batch; Step 2: Set the objective function F of the neural network model as F = a × La + (1-a) × Lo + Pe, and if it is the first round, a is 1; Step 4: When there is a service function chain SFC request, this step is performed in a loop: input the underlying network resource information at this time into the neural network prediction model to obtain the placement position and occupied link; evaluate the occupation scheme by using the objective function to obtain the delay La, the server and link load balancing score Lo of the arrangement scheme, wherein the load balancing condition is represented by the variance of the resource occupation proportion of each server and link; determine whether the arrangement is successful, i.e., whether the computing resources and bandwidth resources in the underlying network can meet the needs of the service function chain SFC; The server probability pre output by the decoder prediction j and the evaluation result F j is stored in the cache, and when the request for processing the service function chain SFC is equal to M, the fifth step is entered; j=0, 1, …, M; Step 5, update the parameters θ of the current trained neural network model k,y ; ； wherein lr is the learning rate of the currently trained neural network model; Step 6: clear the cache and return to Step 3 to continue training; Step 7, when the training round is greater than batch, save the current trained neural network model; change the parameter a = a - 1 / n in the objective function F of the neural network model s , return to step 3 for training until n s neural network models are trained. arrange the service function chain SFC by using the trained neural network prediction model. 2.The deep reinforcement learning based intelligent service function chaining deployment method according to claim 1, characterized in that: The neural network prediction model for predicting the server and link occupied by a virtual network function VNF in a service function chain SFC predicts the occupied server first and then calculates the recommended path connecting the servers when performing prediction. 3.The deep reinforcement learning based intelligent service function chaining deployment method according to claim 2, characterized in that: The model for predicting the occupied server in the neural network prediction model is composed of multiple neural network models with the same structure; The neural network model comprises an input layer, an encoder, a decoder and an output layer; The input layer is used to process and normalize the computing resources, degree, and bandwidth resources of the servers in the underlying network into a 3×n matrix. f The characteristic matrix of , where n f This refers to the number of servers in the underlying network. The encoder is composed of multiple long short-term memory recurrent neural networks (LSTM), and the LSTM extracts the final hidden state c of the bottom layer network through three gate structures t t = 1, 2,..., n f ; The decoder, consisting of multiple LSTMs, takes as input the last hidden state c of the encoder nf and outputs the final after a series of LSTM operations; where, is the probability of selecting the t-th server. The output layer adopts a sampling function to sample the output probability in the decoder to obtain the occupied server. 4.The deep reinforcement learning based intelligent service function chaining deployment method according to claim 3, characterized in that, The long short-term memory recurrent neural network LSTM extracts the final hidden state c of the bottom layer network through three gate structures t The method is as follows: The first gate structure is set as a forget gate, which stores the feature x of this node. t and the output h of the previous LSTM t-1 Perform a non-linear mapping and output a vector f. t =σ(θ f (h f-1 +x t )+b f ), where θ f and b f The weights and offsets of the forget gate structure, f t The value range is between [0,1], representing different degrees of retention, with 0 indicating no retention at all and 1 indicating full retention. The second gate structure is set as an input gate, which first determines the value i to be updated t = σ(θ i (h t-1 +x t )+b i ), and then generates a candidate vector of the state to be added = tanh(θ c (h t-1 +x t )+b c ), and finally updates the cell state c t = f t c t-1 +i t ; wherein θ i and b i are the weights and offsets of the input gate structure, θ c and b c are the weights and offsets of the tanh structure. The third gate structure is configured as an output gate for finally outputting a value h t = σ(θ0(h t-1 + x t ) + b0) tanh(c t ) where θ0and b0are weights and bias of the output gate structure. 5.The deep reinforcement learning based intelligent service function chaining deployment method according to any one of claims 1-4, characterized in that, The recommended path algorithm in the neural network prediction model is as follows: First, select a starting server p i and an ending server p i+1 , and set i to 1. where i = 1, 2,..., n z ; [p1, p2,..., ] are all predicted occupied servers; Second, the origin server p i joins the known set, sets the flag to true, indicating that the shortest path from the source to this point has been determined, and adds each server p m to the unknown set. Third step, updating the origin server p i with the other servers p m the cost dis(p i , p m ); Fourth step, in the unknown set, select dis(p) i ,p m The server with the smallest median value p x , will server p x Add to a known set; Step 5. In the remaining servers, compute dis(p i ,p m ) > dis(p i ,p x ) + dis(p x ,p m ); If this inequality holds, then dis(p i ,p m )=dis(p i ,p x )+dis(p x ,p m ), in which case p i and p m path through p x ; Loop until p i+1 Point to join the known set, get dis(p i ,p i+1 ) is the shortest distance; Step 6: After obtaining the shortest distance between two servers, set i = i + 1 and return to Step 1 to select a new starting server and end server until the shortest paths between all servers are calculated. 6.The deep reinforcement learning based intelligent service function chaining deployment method according to claim 1, characterized in that: When arranging a service function chain SFC, first collect the underlying network resources, then construct a Pareto front based on the collected information, and finally perform greedy selection. 7.The deep-reinforcement learning based intelligent service function chaining deployment method according to claim 6, characterized in that, The method for arranging a service function chain SFC is as follows: Step 1: After receiving a service function chain SFC request, collect the underlying network resource information of the current time slice, and process the information into data convenient for the neural network prediction model to calculate; Step 2: input the collected underlying network resource data into the neural network prediction model to construct a Pareto optimal solution set for this problem: Meanwhile, the bottom-layer network resource data is input into n s neural network models of the neural network prediction model, and n s scheduling methods are obtained s The n s scheduling methods are a Pareto optimal solution set of this problem. Thirdly, find the solution that meets min(Lo) under la≤L from each solution in the Pareto optimal solution set, where L is the maximum allowed time delay of the user. 8.A system for intelligent service function chaining deployment based on deep reinforcement learning, characterized in that, Comprise: a memory; and a processor coupled to the memory, the processor configured to execute instructions stored in the memory to perform the method of claim 1-7. 9.A non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the method of claim 1-7.

Images