Service function chaining parallelization deployment method and system based on reinforcement learning

Abstract

The application provides a service function chain parallelization deployment method and system based on reinforcement learning, which comprises the following steps: (1) modeling a physical network; (2) modeling an SFC request; (3) modeling a deployment process; (4) MDP modeling; (5) dynamic copy generation; (6) dynamic prediction; (7) extracting the relationship characteristics of the physical network, inputting the Actor network and the Critic network for feature learning, and obtaining the strategy action with the maximum occurrence probability to act on the network environment. By introducing the dynamic copy and the dynamic prediction technology, the application decouples the result dependence of the current request on the previous request into the prediction dependence on the previous request processing result, realizes the parallel processing of multiple requests, does not lose the possible dependence relationship between the requests, uniformly deploys the multiple requests to the entire network, and avoids the network unavailability caused by the centralized deployment on individual nodes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of service function chain deployment, and specifically to a service function chain parallelization deployment method and system based on reinforcement learning. BACKGROUND

[0002] Network Function Virtualization (NFV) is a new network architecture technology that decouples traditional network functions (such as firewalls, routers, load balancers, etc.) from dedicated hardware devices and converts them into software form for deployment and management on general-purpose servers. Compared with traditional network function deployment methods, NFV technology has higher flexibility, scalability and cost-effectiveness.

[0003] Traditional network function deployment methods require the use of dedicated hardware devices, and each network function requires an independent physical device, resulting in high deployment and maintenance costs and inflexibility. The emergence of NFV technology enables network functions to run in a virtual manner on general-purpose servers, sharing computing, storage and network resources to achieve centralized management and dynamic deployment of network functions. The core of NFV technology is to virtualize network functions into Virtualized Network Functions (VNF), which are network functions implemented in software form and can be deployed and managed on a virtualization platform. By virtualizing network functions, network functions can be flexibly scaled, quickly deployed and adjusted, improving network flexibility and scalability.

[0004] However, although NFV technology brings many advantages, it also faces some challenges. One of them is how to effectively allocate and manage the computing, storage and network resources required by VNFs. Since different VNFs have different resource requirements, and network traffic load and service quality requirements also change over time, a dynamic resource allocation method is needed to meet the needs of VNFs and dynamically adjust resources according to actual conditions. In addition, factors such as resource utilization and performance indicators need to be considered to achieve efficient operation of network functions.

[0005] With the development of network function virtualization (NFV), service function chaining (SFC) has become an important network architecture concept. SFC refers to connecting multiple network functions (VNFs) in a specific order to meet specific network service requirements. By connecting different VNFs in the order of the service chain, complex network service functions such as traffic filtering, security checking, load balancing, etc. can be realized. However, existing researches mostly focus on serial deployment, which cannot fully utilize hardware resources and is not fast enough in deployment. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of the prior art and provide a service function chain parallel deployment method and system based on reinforcement learning. Unlike other serial deployment methods, the present method can be deployed in parallel, thereby fully utilizing hardware resources and speeding up deployment.

[0007] To achieve the above purpose, the technical solution adopted by the present application is:

[0008] The first aspect of the present application provides a service function chain parallel deployment method based on reinforcement learning, comprising:

[0009] (1) Model the physical network;

[0010] The physical network is represented as an undirected weighted graph G=(V, E), where V is a set of physical servers that can deploy VNFs and forward traffic, E is a set of physical links connecting these servers; u, v∈V represent two physical servers, uv∈E represents the physical link connecting the two servers; each physical server u has CPU maximum capacity and memory maximum capacity Two kinds of allocable resources;

[0011] (2) Model the SFC request;

[0012] Let F represent the set of VNF types, and represent the SFC request as a four-tuple R i ={S i , D i , F i , Ψ i}, where S i , D i represent the source node and target node of request i respectively, F i ∈F represents the set of VNFs required by request i, and Ψ i is the maximum delay that request i can accept; represents the minimum bandwidth that needs to be allocated to the request to meet the delay requirement;

[0013] directed weighted graph G i = (V i , E i ) represents an SFC request, where V i = {S i ∪D i ∪F i}, E i represents a virtual link between nodes and ; each f∈F i has different resource requirements, and respectively represent the CPU and memory numbers required by the corresponding VNFf i of node ;

[0014] (3) Model the deployment process;

[0015] The deployment process is regarded as a mapping relationship, that is, G i = (V i , E i ) → G = (V, E);

[0016] The mapping process is defined as a set of search and selection operations, wherein, is a set of candidate server nodes selected for f k with node χ k (χ1=S) as the starting point, A k represents selecting a server from the candidate set, P k determines whether to select the server selected in the previous step as a deployment node (χ k+1 =A k ; k=k+1) or a forwarding node (χ k =A k ), S represents the source node, and D represents the destination node;

[0017] (4) MDP modeling;

[0018] The service function chain parallel deployment problem is modeled as an MDP five-tuple <S, A, R, P, γ>;

[0019] Wherein, the state S is the observation value of the environment, including the attributes of the resource distribution of each node in the network, the distribution of the link bandwidth; the action space A is the behavior that the agent can take, including all nodes in the network; the reward R is the reward signal received by the agent after state transition and action, and a positive reward is returned for successful deployment, and a negative reward is returned for failed deployment; the transition probability P describes which states will be transferred after taking a certain action in a certain state; the discount factor γ is used to balance the importance of current reward and future reward;

[0020] (5) Dynamic copy generation;

[0021] A state copy is generated according to the current network state each time a request arrives, and subsequent processing of the request and state transition are based on the copy; when the request processing is completed, the number of resources required for processing the request is synchronized to the source network; wherein the synchronization to the source network process and the generation of the state copy use concurrency control;

[0022] (6) Dynamic prediction is performed;

[0023] If two requests R1 and R2 arrive in the same minimum time window δ in sequence, the R1 request is also taken as part of the underlying network state when processing the R2 request, so that the R2 can perceive and predict the processing result of the R1 request;

[0024] (7) The relationship features of the physical network are extracted and input into the Actor network and the Critic network for feature learning to obtain the strategy action with the highest occurrence probability to act on the network environment;

[0025] The convolutional neural network GCN is used to extract the features of the physical network to obtain deeper relationship features between each node;

[0026] In each step, the current network state is input into the GCN to obtain the feature matrix Wherein, l gcn is the dimension of the feature vector extracted by the GCN; the process of extracting the feature by the GCN is as follows:

[0027] Wherein, is a trainable parameter matrix, is the process of extracting the feature by the GCN through convolution, self-connection is realized by adding a unit matrix Λ;

[0028] For the network state the feature is extracted by the GCN to obtain the feature matrix;

[0029] extract features of the physical network G and multiple SFC requests R in the time window δ through GCN δ , to obtain a feature matrix and concatenate the two feature matrices and to obtain two feature vectors, and finally combine the three vectors to obtain the overall state S i , the feature vector of

[0030] After completing the extraction and conversion of the features, the feature vectors are respectively sent to the Actor network and the Critic network;

[0031] In the Actor network, first, two fully connected layers are used to further extract features, and the results are converted to the same dimension as the action space. Then, the Mask layer is used to mask out invalid actions. Finally, the softmax layer is used to convert the results to a probability distribution between 0 and 1. The action a t is obtained by sampling the distribution, and the logprob value of the action is calculated, and finally the policy π = (s t , a t ) is obtained.

[0032] In the Critic network, the feature vector is passed through two fully connected layers and finally converted into a single Q-value.

[0033] Based on the above, the specified time window δ is expanded or reduced according to the traffic size.

[0034] Based on the above, the parallelism of the concurrent control in the dynamic replica generation process is 8.

[0035] Based on the above, when extracting the features of the physical network G and multiple SFC requests R in the time window δ through GCN δ , the features of other requests in the same time window also need to be considered as part of the current features, and the features of the current request are obtained by comprehensive analysis.

[0036] The features of other requests to be considered include CPU, memory, storage space, network bandwidth, latency, node throughput, and node reliability.

[0037] Based on the above, when generating a deployment scheme, if the physical node resources are insufficient to cause the request to fail to be deployed, the node will be marked as "dirty" within a certain time after the scheme is applied. After that, the DRL model will ignore the node and redeploy the current request.

[0038] Based on the above, feature learning is performed in 3 steps, in step 1, the receiving rate of the deployment scheme is focused on; step 2 optimizes the link length and delay of the deployment scheme based on step 1; step 3 focuses on whether to evenly deploy multiple deployment schemes to multiple nodes based on step 2, so as to balance the use of system resources.

[0039] The second aspect of the application provides a service function chain parallelization deployment system based on reinforcement learning, which is used to implement the service function chain parallelization deployment method based on reinforcement learning.

[0040] The deployment system comprises:

[0041] A physical network modeling unit is configured to model the physical network.

[0042] An SFC request modeling unit is configured to model the SFC request.

[0043] A deployment process modeling unit is configured to model the deployment process.

[0044] An MDP modeling unit is configured to model the service function chain parallelization deployment problem as an MDP.

[0045] A dynamic copy generation unit is configured to generate state copies in the deployment process.

[0046] A dynamic prediction unit is configured to enable one of two requests that arrive in sequence within the same minimum time window delta to perceive and predict the processing result of the other request in the deployment process.

[0047] A reinforcement learning network unit is configured to extract the relationship features of the physical network, and input the Actor network and the Critic network for feature learning to obtain a set of policy actions.

[0048] A deployment unit is configured to select the policy action with the maximum probability from the set of policy actions to act on the network environment.

[0049] The third aspect of the application provides a service function chain parallelization deployment apparatus based on reinforcement learning, which comprises:

[0050] A memory; and

[0051] A processor coupled to the memory, the processor being configured to execute the service function chain parallelization deployment method based on reinforcement learning based on instructions stored in the memory.

[0052] The fourth aspect of the application provides a non-transitory computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the service function chain parallelization deployment method based on reinforcement learning.

[0053] The present application has outstanding substantial features and significant progress compared with the prior art. Specifically, the present application decouples the current request's dependence on the result of the previous request into a prediction of the result of the previous request by introducing dynamic copy and dynamic prediction technology, realizes parallel processing of multiple requests, does not lose the possible dependence between requests, deploys multiple requests evenly in the entire network, and avoids network unavailability caused by centralized deployment in individual nodes. The parallelization scheme of the present application can fully utilize hardware resources and accelerate deployment speed. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a schematic diagram of the principle of the neural network model of the present application.

[0055] Figure 2 is the average processing time of requests using different numbers of threads during the experiment. DETAILED DESCRIPTION

[0056] The technical solutions of the present application will be described in further detail below through specific embodiments.

[0057] Embodiment 1

[0058] As shown in Figure 1 , the present embodiment provides a service function chain parallelization deployment method based on reinforcement learning according to the first aspect of the present application, which comprises:

[0059] (1) Model the physical network;

[0060] The physical network is represented as an undirected weighted graph G=(V,, E), where V is a set of physical servers that can deploy VNFs and perform traffic forwarding, E is a set of physical links connecting these servers; u, v∈V represent two physical servers, uv∈E represents a physical link connecting the two servers; each physical server u has two allocable resources, CPU maximum capacity and memory maximum capacity .

[0061] (2) Model the SFC request;

[0062] Let F represent a set of VNF types; each SFC request contains a source node, a target node, a maximum tolerable delay, and a series of VNFs, so the SFC request is represented as a four-tuple R i ={S i , D i , F i , Ψ i}, where S i , D i represent the source node and target node of request i, respectively, and Fi ∈ F represents the set of VNFs that request i needs, Ψ i is the maximum latency that request i can accept; represents the minimum bandwidth that needs to be allocated for the request in order to meet the latency requirement;

[0063] In order to guarantee the order of VNFs, the directed weighted graph G i = (V i , E i ) represents an SFC request, where V i = {S i ∪ D i ∪ F i}, represents a virtual link between nodes and ; each F i has different resource requirements, and respectively represents the CPU and memory required by the VNFf i corresponding to node .

[0064] (3) Model the deployment process;

[0065] The deployment process is regarded as a mapping relationship, that is, G i = (V i , E i ) → G = (V, E) ;

[0066] The mapping process is defined as a set of search and selection operations, where, is the set of candidate server nodes selected for f k starting from node χ k (χ1 = S), A k represents selecting a server from the candidate set, P k determines whether to select the server selected in the last step as a deployment node (χ k+1 = A k ; k = k + 1) or a forwarding node (χ k = A k ), S represents the source node, and D represents the destination node.

[0067] (4) MDP modeling;

[0068] The service function chain parallel deployment problem is modeled as an MDP five-tuple <S, A, R, P, γ> ;

[0069] Wherein, the state S is the observation value of the environment, including the attributes of the resource distribution of each node in the network, the distribution of the link bandwidth; the action space A is the behavior that the agent can take, including all nodes in the network; the reward R is the reward signal received by the agent after state transition and action, and a positive reward is returned for successful deployment, and a negative reward is returned for failed deployment; the transition probability P describes which states will be transferred after taking a certain action in a certain state; the discount factor γ is used to balance the importance of the current reward and the future reward.

[0070] (5) Dynamic copy generation;

[0071] Because the reinforcement learning algorithm needs to rely on the underlying network state when making policy selection, state transition occurs after the action is executed, and then the policy selection of the next action is based on the new state. Therefore, when multiple requests arrive, only the requests can be processed in sequence to ensure that the model selects the correct policy, which will cause the inability to handle the burst traffic in time, causing traffic backlog;

[0072] During deployment, a state copy is generated according to the current network state each time a request arrives, and subsequent processing and state transition of the request are based on this copy. In this way, each request uses an independent state and does not affect each other. When the request is processed, the number of resources required to process the request is synchronized to the source network; wherein the synchronization to the source network process and the generation of the state copy use concurrent control.

[0073] (6) Dynamic prediction;

[0074] The dynamic copy generation technology enables multiple requests to be processed concurrently based on their own state copies, but this will cause multiple requests to make decisions based on the same state copy, resulting in multiple requests being deployed to the same physical node, causing the network physical node to be unavailable;

[0075] Therefore, the dynamic prediction technology is introduced in the deployment process. Assuming that two requests R1 and R2 arrive in the same small time window δ (set to the average processing time of a single request) in sequence, R2 will generate a deployment strategy based on the state after R1 is deployed in the serialization scheme, and in the parallelization scheme, two identical copies will be generated to process the two requests. Therefore, R1 request is also part of the underlying network state when processing R2 request, so that R2 can perceive and predict the processing result of R1 request, thereby avoiding excessive resource competition with R1 request.

[0076] (7) Extract the relationship features of the physical network, and input the Actor network and the Critic network for feature learning to obtain the policy action with the highest occurrence probability to act on the network environment.

[0077] The convolutional neural network GCN is used for feature extraction of the physical network to obtain deeper relationship features between each node.

[0078] In each step, the current network state is sent to the GCN to obtain the feature matrix wherein l gcn is the dimension of the feature vector extracted by the GCN; the process of feature extraction by the GCN is as follows:

[0079] wherein, is a trainable parameter matrix, is the process of feature extraction by the GCN through convolution, and self-connection is achieved by adding a unit matrix Λ;

[0080] For the network state feature extraction is performed by the GCN to obtain the feature matrix.

[0081] The features of the physical network G and multiple SFC requests R δ in the time window δ are extracted by the GCN to obtain the feature matrices and The two feature matrices and are respectively subjected to splicing operations to obtain two feature vectors, and finally three vectors are combined to obtain the feature vector of the overall state S i

[0082] wherein the specified time window δ is expanded or reduced according to the traffic size.

[0083] After the extraction and conversion of the features, the feature vectors are respectively sent to the Actor network and the Critic network.

[0084] In the Actor network, the features are first further extracted through two fully connected layers, and the results are converted to the same dimension as the action space, then the Mask layer is used to shield invalid actions, and finally the softmax layer is used to convert the results to a probability distribution between 0 and 1, the action a t is obtained by sampling the distribution, and the logprob value of the action is calculated, and finally the policy π = (s t , a t ) is obtained.

[0085] In the Critic network, the feature vector is subjected to two fully connected layers, and finally converted into a single Q-value.​

[0086] Specifically, the physical network G and multiple SFC requests R within the time window δ are extracted using GCN. δ When determining the characteristics of a request, it is also necessary to include the characteristics of other requests within the same time window as part of the current characteristics, and then synthesize the characteristics of the current request.

[0087] Other characteristics to consider for requests include CPU, memory, storage space, network bandwidth, latency, node throughput, and node reliability.

[0088] In other embodiments, when a deployment scheme is generated and the application of the scheme fails due to insufficient physical node resources, the node is marked as a "dirty node" for a certain period of time. After that, the DRL model will ignore the node and redeploy the current request. If the node with insufficient resources is not marked as a "dirty node", there is still a high probability that the request will continue to be deployed to the node in the subsequent deployment process, causing the new deployment scheme to fail again.

[0089] In other embodiments, feature learning is performed in three steps, with optimization points of interest being added sequentially in each step. Step 1 focuses on optimizing the request acceptance rate to quickly obtain a model capable of generating an effective deployment plan. Step 2 focuses on optimizing the request acceptance rate and deployment plan latency, aiming to reduce latency while still generating an effective deployment plan. Step 3 focuses on optimizing the request acceptance rate, deployment plan latency, and changes in resource distribution entropy after deployment, ensuring a more even distribution of multiple deployment plans across underlying physical nodes. This avoids concentrated deployment on a single node, which could lead to resource shortages and inability to provide services to the outside world.

[0090] like Figure 2 As shown, the average processing time for requests using different numbers of threads is compared. When the number of threads is 1, the model degrades to serial processing, leading to increased processing time. As the number of worker threads increases, the average waiting time and processing time for requests gradually decrease. However, increasing the number of requests processed simultaneously leads to excessive resource contention, affecting the success rate of the generation strategy and thus increasing processing time. Therefore, there is a balance point for the number of threads. While increasing the number of threads reduces request queuing time and service time, it also increases system overhead and processing time. In summary, in the experimental environment of this invention, the optimal number of threads is 8. At this point, compared to serial processing, the average processing time is reduced by approximately 3-4 times when handling high-concurrency requests.

[0091] Example 2

[0092] The embodiment provides a service function chain parallelization deployment system based on reinforcement learning, which is used for implementing the service function chain parallelization deployment method based on reinforcement learning in the embodiment 1.

[0093] The deployment system comprises:

[0094] A physical network modeling unit is configured to model a physical network.

[0095] An SFC request modeling unit is configured to model an SFC request.

[0096] A deployment process modeling unit is configured to model a deployment process.

[0097] An MDP modeling unit is configured to model a service function chain parallelization deployment problem as an MDP.

[0098] A dynamic replica generation unit is configured to generate a state replica in the deployment process.

[0099] A dynamic prediction unit is configured to enable one of two requests that arrive in a same minimum time window delta in sequence to perceive and predict a processing result of the other request in the deployment process.

[0100] A reinforcement learning network unit is configured to extract relationship features of the physical network, and input the Actor network and the Critic network for feature learning to obtain a policy action set.

[0101] A deployment unit is configured to select a policy action with the maximum probability from the policy action set to act on the network environment.

[0102] Embodiment 3

[0103] The embodiment provides a service function chain parallelization deployment apparatus based on reinforcement learning, which comprises:

[0104] A memory; and

[0105] A processor coupled to the memory, the processor being configured to execute the service function chain parallelization deployment method based on reinforcement learning in the embodiment 1 based on instructions stored in the memory.

[0106] Embodiment 4

[0107] The embodiment provides a non-transitory computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the service function chain parallelization deployment method based on reinforcement learning in the embodiment 1.

[0108] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer non-transitory readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) having computer readable program code embodied in the medium.

[0109] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It should be understood that each flow and / or block in the flowchart and / or block diagrams, and a combination of flows and / or blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate a means for implementing the functions specified in the flowchart and / or block diagrams of the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flow or flows and / or block or blocks.

[0110] These computer program instructions can also be stored in a computer readable storage medium that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a manufacture product including instruction means, which implement the functions specified in the flowchart and / or block diagrams of the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flow or flows and / or block or blocks.

[0111] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowchart and / or block diagrams of the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flow or flows and / or block or blocks.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it; although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or some technical features can be replaced by equivalent; without departing from the spirit of the technical solutions of the present application, they should be covered in the technical solution range of the present application claimed.

Claims

1. A method for service function chaining parallelization deployment based on reinforcement learning, characterized in that, Comprise: (1) modeling the physical network; The physical network is represented as an undirected weighted graph G = (V, E), where V is a set of physical servers on which VNFs can be deployed and traffic is forwarded, and E is a set of physical links connecting the servers; u, v ∈ V represent two physical servers, and uv ∈ E represents a physical link connecting the two servers; each physical server u has a CPU maximum capacity and a memory maximum capacity Two allocable resources; (2) modeling the SFC request; Let F denote the set of VNF types, and let R denote the SFC request as a four-tuple i = {S i , D i , F i , Ψ i}, where S i , D i denote the source and destination nodes of request i, respectively, F i ∈ F denotes the set of VNFs required by request i, and Ψ i is the maximum latency that request i can accept; denotes the minimum bandwidth that needs to be allocated to the request in order to meet the latency requirement; Directed and weighted graph G i = (V i , E i ) represents an SFC request, where V i = {S i ∪ D i ∪ F i}, represents a virtual link between nodes and ; each f∈F i has different resource requirements, with representing the number of CPUs and memory required by the corresponding VNFf for node , respectively; (3) modeling the deployment process; The deployment process is considered as a mapping relation, i.e. G i = (V i , E i ) → G = (V, E); The mapping process is defined as a set of search and selection operations, wherein, is the set of candidate server nodes, k (χ1=S) as the starting point for f k selecting a set of candidate server nodes, A k denotes the selection of a server from the candidate set, P k determining whether the server selected in the previous step is to be a deployment node (χ k+1 =A k ; k=k+1) or a forwarding node (χ k =A k ), S denotes the source node, and D denotes the destination node; (4) MDP modeling; Model the service function chain parallelization deployment problem as a MDP five-tuple <S, A, R, P, γ>; Wherein, the state S is the observation value of the environment, including the attributes of the resource distribution of each node in the network, the distribution of the link bandwidth; the action space A is the behavior that the agent can take, including all nodes in the network; the reward R is the reward signal received by the agent after state transition and action, and a positive reward is returned for successful deployment, and a negative reward is returned for failed deployment; the transition probability P describes which states will be transferred after taking a certain action in a certain state; the discount factor γ is used to balance the importance of the current reward and the future reward; (5) dynamic replica generation; A state replica is generated according to the current state of the network each time a request arrives, and the subsequent processing of the request and state transition are based on the replica, and when the request processing is completed, the number of resources required for processing the request is synchronized to the source network; wherein the synchronization to the source network process and the generation of the state replica use concurrency control; (6) dynamic prediction; If two requests R1 and R2 arrive in the same minimum time window δ, R1 request is also taken as part of the underlying network state when processing R2 request, so that R2 can perceive and predict the processing result of R1 request; (7) extract the relationship features of the physical network, and input the Actor network and the Critic network for feature learning to obtain the strategy action with the highest occurrence probability to act on the network environment; Convolutional neural network GCN is used to extract features of the physical network to obtain deeper relationship features between each node; In each step, the current network state is sent into the GCN to get the feature matrix where l gcn is the dimension of the feature vector extracted by the GCN; the process of the GCN extracting features is as follows: wherein, is a trainable parameter matrix, is the process of feature extraction by convolution of the GCN, self-connection is achieved by adding a unit matrix Λ; For network status Feature extraction is performed by the GCN to obtain a feature matrix; extracting physical network G and multiple SFC requests R in time window δ by GCN δ obtaining feature matrix and performing splicing operation on two feature matrices and respectively to obtain two feature vectors, and finally combining three vectors to obtain overall state S i feature vector of After the extraction and conversion of the features are completed, the feature vectors are respectively sent into the Actor network and the Critic network; In the Actor network, first, the features are further extracted through two fully connected layers, and the results are converted to the same dimension as the action space. Then, the Mask layer is used to mask out the invalid actions. Finally, the softmax layer is used to convert the results to a probability distribution between 0 and 1. The action a is obtained by sampling the distribution, and the logprob value of the action is calculated. Finally, the policy π = (s, a) is obtained. t t t ​​​ In the Critic network, the feature vector After two fully connected layers, it is finally transformed into a single Q-value. 2.The method of claim 1, wherein: The specified time window δ is expanded or reduced according to the traffic size. 3.The method of claim 1, wherein: The parallelism of the concurrency control in the dynamic replica generation process is 8. 4.The method of claim 1, wherein: extracting the physical network G and a plurality of SFC requests R within a time window δ by GCN δ When the feature of the current request is obtained, the features of other requests within the same time window are also needed as part of the current feature, and the features of the current request are obtained comprehensively. Other characteristic attributes to be considered for the request include CPU, memory, storage space, network bandwidth, latency, node throughput, and node reliability. 5.The method of claim 1, wherein: When a deployment scheme is generated, if the physical node resources are insufficient to cause the request to fail after the scheme is applied, the node is marked as "dirty" within a certain time, and the DRL model will ignore the node and redeploy the current request. 6.The method of claim 1, wherein: Feature learning is performed in 3 steps, in step 1, the acceptance rate of the deployment scheme is focused on; in step 2, the link length and latency of the deployment scheme are optimized based on step 1; in step 3, whether multiple deployment schemes are evenly deployed to multiple nodes is focused on based on step 2, so that the system resources are balanced. 7.A system for service function chaining parallelization deployment based on reinforcement learning, characterized in that, A reinforcement learning-based service function chain parallelization deployment method according to any one of claims 1-6; The deployment system comprises: a physical network modeling unit for modeling the physical network; an SFC request modeling unit for modeling the SFC request; a deployment process modeling unit for modeling the deployment process; an MDP modeling unit for MDP modeling of the service function chain parallelization deployment problem; a dynamic copy generation unit for generating a state copy in a deployment process; a dynamic prediction unit for enabling one of two requests that arrive in sequence within the same minimal time window δ to perceive and predict the processing result of the other request in the deployment process; a reinforcement learning network unit for extracting relationship features of a physical network and inputting the Actor network and the Critic network for feature learning to obtain a policy action set; a deployment unit for selecting a policy action with the maximum probability from the policy action set to act on the network environment.

8. A service function chaining parallelization deployment apparatus based on reinforcement learning, characterized by, comprising: a memory; and a processor coupled to the memory, the processor configured to execute instructions stored in the memory to perform the method of claim 1-6. 9.A non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the method of claim 1-6.

Images