A service function chaining placement method based on multi-objective deep reinforcement learning

Abstract

The application discloses a service function chain placement method based on multi-target deep reinforcement learning, specifically comprising the following steps: generating a service function chain instance to be placed and user service demand; mapping the user service demand into sub-problems, and assigning a weight preference to each optimization index; judging whether the weight preference appears at a previous time step, and retrieving an optimal neural network model corresponding to the problem of the current weight preference; otherwise, initializing the neural network model under the current weight preference; acquiring current physical network environment information and inputting the information into a space-time encoder for information coding; inputting the coded information into a distributed reinforcement learning intelligent agent, and performing decision calculation and generating a decision function through an intelligent agent neural network; saving and updating the neural network model parameters under the current weight, and performing joint decision of all intelligent agents in the distributed reinforcement learning; and completing the service function chain placement task. The application improves the resource allocation capability of placing the service function chain under different dynamic network environments.

Description

Technical Field

[0001] This invention pertains to resource allocation technology in dynamic network environments, and particularly relates to a method for placing service function chains based on multi-objective deep reinforcement learning. Background Technology

[0002] Network Functions Virtualization (NFV) is a technology that can quickly and flexibly meet the Quality of Service (QoS) requirements of 5G services. It decomposes network services into a sequence of Virtual Network Functions (VNFs) and strings them together to form Service Function Chains (SFCs). The optimal resource allocation required to deploy SFCs on the network virtualization infrastructure, while meeting user QoS requirements, is defined as the SFC placement problem.

[0003] Current research has yet to provide a sufficiently effective solution to the SFC (Service Controller Placement) problem in mobile edge computing, leading to resource waste. Existing methods mainly fall into two categories: single-objective optimization methods, which consider only one criterion (such as latency), are simple to model but have limited applicability; and multi-objective optimization methods, which consider multiple conflicting or influencing criteria (such as latency and network congestion), are complex to model but have wide applicability. However, modeling and solving multi-objective optimization problems face significant challenges, primarily because the increased number of nodes and services in the network reduces model scalability and complicates finding the optimal solution.

[0004] To solve the multi-objective problem, there are two main types of existing methods: one is based on evolutionary algorithms or heuristic algorithms. These methods are limited in high-dimensional or continuous spaces, have large computational overhead, and are difficult to adapt to dynamic environments. The other is deep reinforcement learning (DRL) algorithms. These methods can effectively utilize environmental information and feedback, update policies through the trade-off between exploration and utilization, and process the iteration of the space through neural networks. This type of multi-objective DRL method is the mainstream of current research. Scholars have made various improvements to it, mainly in two directions: one is to improve it through emerging machine learning models. For example, Zhang et al. [1] proposed an architecture that uses the Transformer model to encode the state information of the input DRL agent. The other is to improve it by increasing the number of DRL agents and redefining the interaction between agents. For example, Wang et al. [2] proposed a distributed architecture called MRPDeep, which assigns an optimization objective to each agent.

[0005] Current technologies only consider single-objective optimization, failing to model multi-objective problems for the placement of service function chains (SFCs). Multi-objective problem modeling is more relevant to real-world applications. However, it doesn't combine various improvement methods, such as integrating emerging machine learning techniques with distributed reinforcement learning. Furthermore, applying DRL for multi-objective optimization in the SFC placement problem requires addressing three challenges: 1) reducing the interference of different weight preferences on agent training; 2) fully exploiting the spatiotemporal correlations in network data; and 3) improving the universality of the algorithm framework.

[0006] References:

[0007] [1]Zhang K, Lin

[0008] [2]Wang Q, Liu J, Liu C, et al.MPRdeep:Multi-Objective Joint OptimalNode Positioning and Resource Allocation for FANETs with Deep Reinforcementlearning[C] / / 2021IEEE 46th Conference on Local Computer Networks(LCN).IEEE,2021:315-318.

[0009] [3]Chen Y.Convolutional neural network for sentence classification[D].University of Waterloo,2015.

[0010] [4]Vaswani A,Shazeer N,Parmar N,et al.Attention is all you need[J].Advances in neural information processing systems,2017,30.

[0011] [5]Chung J,Gulcehre C,Cho KH,et al.Empirical evaluation of gatedrecurrent neural networks on sequence modeling[J].arXiv preprint arXiv:1412.3555,2014. Summary of the Invention

[0012] To address the above problems, this invention provides a service function chain placement method based on multi-objective deep reinforcement learning.

[0013] The present invention provides a service function chain placement method based on multi-objective deep reinforcement learning, comprising the following steps:

[0014] Step 1: Generate the Service Function Chain (SFC) instance to be placed, as well as the user service requirements.

[0015] Step 2: Pre-training with equal weights: Map user service requirements to sub-problems and assign a weight preference to each optimization indicator; based on the assigned weight preference combination, search in the historical training records to see if the current weight has been trained; if it has been trained, directly extract the optimal neural network model of the current weight combination in the previous time; otherwise, initialize the neural network model of the current weight combination.

[0016] Step 3: Obtain current physical network environment information.

[0017] Step 4: Spatiotemporal coding: Input the acquired environmental information into the spatiotemporal encoder for information encoding.

[0018] Step 5: Distributed reinforcement learning: Input the encoded information as the state into the DDRL-STE agent, the agent's neural network performs decision calculations and generates a decision function; save and update the neural network model parameters under the current weights, and execute the joint decision-making of all agents in DDRL-STE.

[0019] Step 6: Service Function Chain (SFC) placement task completed.

[0020] The above-mentioned equal-weight pre-training specifically refers to:

[0021] Different Service Function Chains (SFCs) are defined as multiple subproblems I with different weights, denoted by the character i∈[1,|I|] for the subproblem of the i-th weight combination; each subproblem is modeled as a neural network, through θ i To represent its network parameters, and This indicates that the optimal subproblem has been found.

[0022] First, map the tasks to different weights. Subproblem i; then, searching the trained neural network library for the existence of pre-trained neural network parameters. If it exists, load it directly. Continue training; if not, initialize the network parameters θ at the current time step. i,k,t Used for training; finally, after all the sequential decisions for subproblem i have been completed, a network model with the maximum long-term expected cumulative reward LTECR can be obtained. Store it.

[0023] Furthermore, the spatio-temporal coding employs a spatio-temporal encoder (STE).

[0024] Input to the Space-Time Encoder (STE): For the placement subproblem of each SFCs, represent the set of all service requests as follows: in This represents a subproblem where the r-th SFC to be placed in the total service request set belongs to the s-th class of SFCs and is mapped to the weight combination of the i-th class; and each service request consists of VNFs, i.e. in This represents the o-th VNF in the s-th SFC within the subproblem of the i-th weight combination; at time step t ∈ [1, |T|], the set of node resources in the physical network is represented as... n∈[1,|N|]; similarly, the set of link resources is represented as B. t ={B 1,t ,...,B l,t}, l∈[1,|L|]; at time step t-1, the physical node where the VNF is placed is represented as n t-1 STE input X t The definition is as follows:

[0025]

[0026] The Structural Design of the Spatiotemporal Encoder STE: The STE structure consists of three layers: a one-dimensional convolutional neural network layer, a Transformer encoding layer, and a gated recurrent unit layer.

[0027] 1) One-dimensional convolutional neural network layer: Using a one-dimensional convolutional neural network layer preserves X t While showcasing temporal structure features, the number of training parameters and computational cost are reduced; the output after two one-dimensional convolutional layers is OC. t The definition is shown in formula (2), where σ is the ReLU activation function, and b1 and b2 are the biases. and There are two learnable weight matrices.

[0028]

[0029] 2) Transformer Encoding Layer: Employing a Transformer encoder enables parallel computation of the input sequence and full-position information processing, improving model training efficiency; the output OE after passing through the Transformer encoding layer... t The definition is shown in formula (3), where Layer MH For multi-head attention function layers, Layer FF This is a feedforward network function layer.

[0030] OE t =Layer FF (Layer MH (OC t )+OC t )+Layer MH (OC t (3)

[0031] 3) Gated Recurrent Unit Layer: The gating mechanism is used to adjust the position weights of the encoded sequence, and its hidden state h is used... t Provides more compact and efficient features for the output of the Transformer encoder; where h t The update is shown in formula (4). Let z be the learnable weight matrix, and z be the update gate signal; the output OG after passing through the gated recurrent unit layer. t The definition is shown in formula (5);

[0032]

[0033] OG t =GRU(OE t ,h t-1 (5)

[0034] Output of the spacetime encoder STE: STE output X′ t Dimensions and input X t The dimensions are exactly the same, that is

[0035] Distributed reinforcement learning includes the formulation of multi-objective Markov decision processes and distributed reinforcement learning structures.

[0036] Multi-objective Markov decision process:

[0037] A multi-objective Markov decision process (MOMDP) ​​is represented by tuples as follows: in Representing the state space, Represents the action space. Represents the state transition probability matrix. For a collection of reward functions, Let f represent the weight preference space, and let f be the function that maps policy π to scalar values.

[0038] 1) State: State It is the collection of physical network information and network service requirements that all intelligent agents can perceive, and the state information X after STE encoding. t ' is the input information for the agent, that is, the state information of the agent input at time step t. It can be represented as

[0039]

[0040] 2) Actions: In the face of continuously and dynamically changing network states, actions are... Defined as discrete; at the current time step t, the decision is made to place the VNF. Physical network infrastructure node n t Let this be denoted as action a. t , Its definition is shown in formula (7).

[0041]

[0042] 3) Rewards: It is the set of reward functions for all agents; in the state When using action a t The intelligent system can obtain rewards by interacting with the network environment. The reward value is used to evaluate whether an action is worth choosing; here, the reward value is set to negative, and the reward... It consists of two parts: the first part is the reward for the legality of VNF placement, and the second part is the reward for the consumption incurred during VNF placement; the reward for the legality of placement is denoted as... Before placement, it is necessary to determine the physical network node n. t Remaining resources Does the current VNF ​​request resource satisfy the current request, and (n t-1 ,n t Remaining bandwidth resources of inter-node links Does the SFC's requested resource meet the requirements? If both are met, then the current action a is considered successful. t If it's valid, the reward value is 0; otherwise, the reward value will be... Setting it to a minimum value terminates the current VNF ​​placement action and resets the previous SFC. VNF already placed in Release, indicating a request to deny; if action a t If it is valid, then the reward for resource consumption during placement needs to be further calculated, i.e., the consumption of the optimization objective. This reward will vary depending on the agent. Specifically, the agents involved have different optimization objectives. According to the optimization objectives defined above, these are the end-to-end latency D. total With network congestion rate U res It will reward value They are and Its definition is shown in formula (8).

[0043]

[0044] Reward value obtained by the agent The definition is shown in formula (9);

[0045]

[0046] 4) Policy Update: In MOMDP, a policy is a mapping from a specific state to an action. The agent learns the optimal strategy by interacting with the corresponding environment in order to obtain LTECR.

[0047] The action value function of strategy π is used This indicates that at time t, the state is... Take action next time t The value of is defined as shown in formula (10), following the expected return of strategy π.

[0048]

[0049] Combine weights and experience Together, they are stored in the agent's experience pool, where the Q-network is used to approximate the Q-function of the target k, and the state... The Q-value vector of the optimization objective k is represented as follows: Its definition is shown in formula (11), where Assign it a preference weight.

[0050]

[0051] For the update of the Q function, as defined in formula (12), α is the step size scalar. Let θ be the objective Q function, defined as shown in formula (13), where θ t and θ t ′ represent the network parameter weights of the Q-value function and the network parameter weights of the target Q-value function, respectively;

[0052]

[0053]

[0054] Distributed reinforcement learning architecture:

[0055] The distributed reinforcement learning structure sets up k agents and considers two optimization objectives: end-to-end latency and network congestion rate, i.e., k∈[0,1]. The design of the entire distributed structure has two core elements: Q-value and action remapping, dynamic weight mechanism and diverse experience playback.

[0056] 1) Remapping of Q-values ​​to actions:

[0057] The z-score normalization method was used to... Standardization is performed, and its definition is shown in formula (14), where μ represents the Q-value vector. The mean is σ, and the standard deviation is defined as shown in formulas (15) and (16).

[0058]

[0059]

[0060]

[0061] 2) Dynamic weighting mechanism and diversity experience playback:

[0062] Map different service request tasks to SFC placement subproblems i with different weights, and assign them specific weight combinations. The weighted sum of the values ​​in the weighted combination is 1. It represents the degree of user preference for different optimization objectives k in the current subproblem i. The joint Q-value vector is defined as shown in formula (17).

[0063]

[0064] The agent selects actions through an ε-greedy policy, which is defined as shown in formula (18).

[0065]

[0066] Where prob. is the abbreviation for probability, and ε, ε∈[0,1] are the probabilities of random exploration and annealing with time step.

[0067] The beneficial technical effects of this invention are as follows:

[0068] This invention proposes a pre-training method with equal weights, which uses the optimization results of the previous step as the initial conditions for the next step, reducing the interference of weight changes; it designs a sequence spatiotemporal coding mechanism to extract the relationship between service requests and network resource information; it adopts a distributed deep reinforcement learning structure, in which each agent observes only one target, optimizes relatively independently, and dynamically adapts to the network and service requests; and it improves the resource allocation capability of placing service function chains in different dynamic network environments. Attached Figure Description

[0069] Figure 1 This is a flowchart of the service function chain placement method based on multi-objective deep reinforcement learning of the present invention.

[0070] Figure 2 This is a conceptual diagram of the present invention.

[0071] Figure 3 A conceptual flowchart for the equal-weight pre-training mechanism.

[0072] Figure 4 This is a structural diagram of the sequence spatiotemporal coding mechanism.

[0073] Figure 5 This is a diagram of the internal structure of the intelligent agent. Detailed Implementation

[0074] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0075] A flowchart of a service function chain placement method based on multi-objective deep reinforcement learning according to the present invention is shown below. Figure 1 As shown, it includes the following steps:

[0076] Step 1: Generate the Service Function Chain (SFC) instance to be placed, as well as the user service requirements (such as low latency and low cost, low latency and high cost, etc.).

[0077] Step 2: Pre-training with equal weights: Map user service requirements to sub-problems and assign a weight preference to each optimization metric (latency, overhead, load balancing, etc.); based on the assigned weight preference combination, search the historical training records to see if the current weight has been trained; if it has been trained, directly extract the optimal neural network model of the current weight combination in the previous time; otherwise, initialize the neural network model of the current weight combination.

[0078] Step 3: Obtain current physical network environment information.

[0079] Step 4: Spatiotemporal coding: Input the acquired environmental information into the spatiotemporal encoder for information encoding.

[0080] Step 5: Distributed reinforcement learning: Input the encoded information as the state into the DDRL-STE agent, the agent's neural network performs decision calculations and generates a decision function; save and update the neural network model parameters under the current weights, and execute the joint decision-making of all agents in DDRL-STE.

[0081] Step 6: Service Function Chain (SFC) placement task completed.

[0082] This invention comprises three core designs: an equal-weight pre-training mechanism, a spatiotemporal encoder (STE) structure, and a distributed reinforcement learning structure. The overall conceptual diagram is shown below. Figure 2 As shown.

[0083] Equal-weight pre-training mechanism:

[0084] Using a scalarization method, different Service Function Chains (SFCs) are defined as multiple subproblems I with different weights, denoted by the characters i∈[1,|I|] to represent the subproblem of the i-th weight combination; each subproblem is modeled as a neural network, through θ i To represent its network parameters, and This indicates that the optimal subproblem has been found.

[0085] First, map the tasks to different weights. Subproblem i; then, searching the trained neural network library for the existence of pre-trained neural network parameters. If it exists, load it directly. Continue training; if not, initialize the network parameters θ at the current time step. i,k,t Used for training; finally, after all the sequential decisions for subproblem i have been completed, a network model with the maximum long-term expected cumulative reward LTECR can be obtained. Storage is performed, the process is as follows: Figure 3 As shown.

[0086] The Space-Time Encoder (STE) will be discussed in detail from three aspects: input, structure, and output.

[0087] Input to the Space-Time Encoder (STE): For the placement subproblem of each SFCs, represent the set of all service requests as follows: in This represents a subproblem where the r-th SFC to be placed in the total service request set belongs to the s-th class of SFCs and is mapped to the weight combination of the i-th class; and each service request consists of VNFs, i.e. in This represents the o-th VNF in the s-th SFC within the subproblem of the i-th weight combination; at time step t ∈ [1, |T|], the set of node resources in the physical network is represented as... n∈[1,|N|]; similarly, the set of link resources is represented as B. t ={B 1,t ,...,B l,t}, l∈[1,|L|]; at time step t-1, the physical node where the VNF is placed is represented as n t-1 STE input X t The definition is as follows:

[0088]

[0089] The spatiotemporal encoder STE has a three-layer structure: a one-dimensional convolutional neural network layer, a Transformer encoding layer, and a gated recurrent unit layer. Figure 4 As shown below, a detailed introduction will follow.

[0090] 1) One-dimensional convolutional neural network layer: Inspired by TextCNN[3], a one-dimensional convolutional neural network layer is used to preserve X. t While showcasing temporal structure features, the number of training parameters and computational cost are reduced; the output after two one-dimensional convolutional layers is OC. t The definition is shown in formula (2), where σ is the ReLU activation function, and b1 and b2 are the biases. and There are two learnable weight matrices.

[0091]

[0092] 2) Transformer Encoding Layer: Inspired by the multi-head attention mechanism in Transformer[4], a Transformer encoder is used to realize parallel computation of the input sequence and full position information processing, thereby improving the model training efficiency; the output OE after the Transformer encoding layer t The definition is shown in formula (3), where Layer MH For multi-head attention function layers, Layer FF For the feedforward network function layer, the expression of the bias has been omitted for the sake of simplicity in the formula notation.

[0093] OE t =Layer FF (Layer MH (OC t )+OC t )+Layer MH (OC t(3)

[0094] 3) Gated Recurrent Unit Layer: Referring to GRU[5], the position weights of the encoded sequence are adjusted using a gating mechanism, and their hidden states h are used. t Provides more compact and efficient features for the output of the Transformer encoder; where h t The update is shown in formula (4). Let z be the learnable weight matrix, and z be the update gate signal; the output OG after passing through the gated recurrent unit layer. t The definition is shown in formula (5);

[0095]

[0096] OG t =GRU(OE t ,h t-1 (5)

[0097] Output of the spacetime encoder STE: STE output X t The dimension of ′ and the input X t The dimensions are exactly the same, that is

[0098] Distributed reinforcement learning includes the formulation of multi-objective Markov decision processes and distributed reinforcement learning structures.

[0099] Multi-objective Markov decision process:

[0100] A multi-objective Markov decision process (MOMDP) ​​is represented by tuples as follows: in Representing the state space, Represents the action space, Represents the state transition probability matrix. For a collection of reward functions, Let f represent the weight preference space, and let f be the function that maps policy π to scalar values.

[0101] 1) State: It is the collection of physical network information and network service requirements that all intelligent agents can perceive, and the state information X′ encoded by STE. t The input information for the agent is the state information of the agent at time step t. It can be represented as

[0102]

[0103] 2) Action: In the face of continuously and dynamically changing network states, actions are... Defined as discrete; at the current time step t, the decision is made to place the VNF. Physical network infrastructure node n t Let this be denoted as action a. t , Its definition is shown in formula (7).

[0104]

[0105] Sequential execution of decisions can guarantee SFC VNFs can be placed in an orderly manner within the physical network. State information. After STE processing, the data is used as input to the agent's neural network. Each agent structure is designed based on the D3QN structure, which can better distinguish the importance of different states and the relative merits of different actions, while reducing the problem of overestimation of Q-values ​​in DQN networks and ensuring the stability of learning. The output is a Q-value vector with the same dimension as the action, representing the probability estimate of the possible actions.

[0106] 3) Reward: It is the set of reward functions for all agents; in the state When using action a t The intelligent system can obtain rewards by interacting with the network environment. The reward value is used to evaluate whether an action is worthwhile; to enable the agent to learn effectively and obtain the maximum long-term expected cumulative reward (LTECR), this paper sets the reward value to negative. It consists of two parts: the first part is the reward for the legality of VNF placement, and the second part is the reward for the consumption incurred during VNF placement; the reward for the legality of placement is denoted as... Before placement, it is necessary to determine the physical network node n. t Remaining resources Does the current VNF ​​request resource satisfy the current request, and (n t-1 ,n t Remaining bandwidth resources of inter-node links Does the SFC's requested resource meet the requirements? If both are met, then the current action a is considered successful. t If it's valid, the reward value is 0; otherwise, the reward value will be... Setting it to a minimum value terminates the current VNF ​​placement action and resets the previous SFC. VNF already placed in Release, indicating a request to deny; if action a tIf it is valid, then the reward for resource consumption during placement needs to be further calculated, i.e., the consumption of the optimization objective. This reward will vary depending on the agent. Specifically, the agents involved have different optimization objectives. According to the optimization objectives defined above, these are the end-to-end latency D. total With network congestion rate U res It will reward value They are and Its definition is shown in formula (8).

[0107]

[0108] Reward value obtained by the agent The definition is shown in formula (9);

[0109]

[0110] 4) Policy Update: In MOMDP, a policy is a mapping from a specific state to an action. The agent learns the optimal strategy by interacting with the corresponding environment in order to obtain LTECR.

[0111] The action value function of strategy π is used This indicates that at time t, the state is... Take action next time t The value of is defined as shown in formula (10), following the expected return of strategy π.

[0112]

[0113] Each agent architecture is designed based on D3QN, but the Q-function is difficult to converge during training due to frequently changing weight values. The weights are combined... and experience Together, they are stored in the agent's experience pool, where the Q-network is used to approximate the Q-function of the target k, and the state... The Q-value vector of the optimization objective k is represented as follows: Its definition is shown in formula (11), where Assign it a preference weight.

[0114]

[0115] For the update of the Q function, as defined in formula (12), α is the step size scalar. Let θ be the objective Q function, defined as shown in formula (13), where θ t and θ′ tLet represent the network parameter weights of the Q-value function and the network parameter weights of the target Q-value function, respectively. This stochastic gradient descent update method makes... Always towards The target direction has been updated to reduce the possibility of overvaluation in standard DQN.

[0116]

[0117]

[0118] Distributed reinforcement learning architecture:

[0119] The distributed reinforcement learning architecture sets up k agents (the internal structure of the agents is as follows) Figure 5 As shown in the figure, two optimization objectives are considered: end-to-end latency and network congestion rate, i.e., k∈[0,1]. The design of the entire distributed structure has two core elements: Q-value and action remapping, dynamic weighting mechanism and diversity experience playback.

[0120] 1) Remapping of Q-values ​​to actions:

[0121] In order to find the optimal action a under the premise of optimizing the objective contradiction. t This paper does not find the optimal action through mapping to a specific Q-value function. Instead, it employs the z-score normalization method. Standardization is performed, and its definition is shown in formula (14), where μ represents the Q-value vector. The mean is σ, and the standard deviation is defined as shown in formulas (15) and (16).

[0122]

[0123]

[0124]

[0125] 2) Dynamic weighting mechanism and diversity experience playback:

[0126] Due to the dynamic changes in network environment and service requirements, a single weight combination cannot be used to assign weights to the output of each agent when modeling multi-objective problems in SFC (System-Focused Function) scenarios. A trade-off is made. Different service request tasks are mapped to SFC placement subproblems i with different weights, and specific weight combinations are assigned to them. The weighted sum of the values ​​in the weighted combination is 1. It represents the degree of user preference for different optimization objectives k in the current subproblem i. The joint Q-value vector is defined as shown in formula (17).

[0127]

[0128] The agent selects actions through an ε-greedy policy, which is defined as shown in formula (18).

[0129]

[0130] Where prob. is the abbreviation for probability, and ε, ε∈[0,1] are the probabilities of random exploration and annealing with time step.

Claims

1. A service function chain placement method based on multi-objective deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Generate the Service Function Chain (SFC) instance to be placed, as well as the user service requirements; Step 2: Pre-training with equal weights: Map user service requirements to sub-problems and assign a weight preference to each optimization metric; Based on the assigned weight preference combination, search the historical training records to see if the current weight combination has already been trained. If the model has already been trained, the optimal neural network model of the current weight combination in the previous time step is directly extracted and used for decision calculation in subsequent distributed reinforcement learning; otherwise, the neural network model of the current weight combination is initialized and used for reinforcement learning training under the current weight combination. Step 3: Obtain current physical network environment information; Step 4: Spatiotemporal coding: Input the acquired environmental information into the spatiotemporal encoder for information encoding; Step 5: Distributed Reinforcement Learning: Distributed reinforcement learning includes the formulation of a multi-objective Markov decision process and the structure of a distributed reinforcement learning system. The encoded information is input as the state into the DDRL-STE agent. The agent's neural network performs decision calculations and generates a decision function. In MOMDP, the policy is a specific mapping π: S→A from state to action. The agent learns the optimal policy by interacting with the corresponding environment to obtain the maximum long-term expected cumulative reward (LTECR). The action is the action used to place the VNF at the current time step t. Physical network infrastructure node n t , where f i s,o Represents the o-th VNF in the s-th SFC of the i-th weight combination subproblem; save and update the neural network model parameters under the current weights, and execute the joint decision of all agents in DDRL-STE; store the network model parameters that can obtain the maximum long-term expected cumulative reward LTECR; Step 6: Service Function Chain (SFC) placement task completed.

2. The service function chain placement method based on multi-objective deep reinforcement learning according to claim 1, characterized in that, The same-weight pre-training specifically refers to: Different Service Function Chains (SFCs) are defined as multiple subproblems I with different weights, denoted by the character i∈[1,|I|] for the subproblem of the i-th weight combination; each subproblem is modeled as a neural network, through θ i To represent its network parameters, and This indicates the optimal subproblem found. First, map the tasks to different weights. Subproblem i; then, searching the trained neural network library for the existence of pre-trained neural network parameters. If it exists, load it directly. Continue training; if not, initialize the network parameters θ at the current time step. i,k,t Used for training; finally, after all the sequential decisions for subproblem i have been completed, a network model with the maximum long-term expected cumulative reward LTECR can be obtained. Store it.

3. The service function chain placement method based on multi-objective deep reinforcement learning according to claim 2, characterized in that, The spatiotemporal coding employs a spatiotemporal encoder, STE. Input to the Space-Time Encoder (STE): For the placement subproblem of each SFCs, represent the set of all service requests as follows: in This represents a subproblem where the r-th SFC to be placed in the total service request set belongs to the s-th class of SFCs and is mapped to the weight combination of the i-th class; and each service request consists of VNFs, i.e. Where f i s,o This represents the o-th VNF in the s-th SFC within the subproblem of the i-th weight combination; at time step t ∈ [1, |T|], the set of node resources in the physical network is represented as... n∈[1,|N|]; similarly, the set of link resources is represented as B. t ={B 1,t ,...,B l,t }, l∈[1,|L|]; at time step t-1, the physical node where the VNF is placed is represented as n t-1 STE input X t The definition is as follows: The structure design of the space-time encoder STE: The structure of STE is divided into three layers, namely a one-dimensional convolutional neural network layer, a Transformer coding layer, and a gated recurrent unit layer; 1) One-dimensional convolutional neural network layer: Using a one-dimensional convolutional neural network layer preserves X t While showcasing temporal structure features, the number of training parameters and computational cost are reduced; the output after two one-dimensional convolutional layers is OC. t The definition is shown in formula (2), where σ is the ReLU activation function, and b1 and b2 are the biases. and There are two learnable weight matrices; 2) Transformer Encoding Layer: Employing a Transformer encoder enables parallel computation of the input sequence and full-position information processing, improving model training efficiency; the output OE after passing through the Transformer encoding layer... t The definition is shown in formula (3), where Layer MH For multi-head attention function layers, Layer FF For feedforward network function layers; OE t =Layer FF (Layer MH (OC t )+OC t )+Layer MH (OC t ) (3) 3) Gated Recurrent Unit Layer: The gating mechanism is used to adjust the position weights of the encoded sequence, and its hidden state h is used... t Provides more compact and efficient features for the output of the Transformer encoder; where h t The update is shown in formula (4). Let z be the learnable weight matrix, and z be the update gate signal; The output OG after passing through the gated loop unit layer t The definition is shown in formula (5); AND t =GRU(OE t ,h t-1 ) (5) Output of the spacetime encoder STE: STE output X′ t Dimensions and input X t The dimensions are exactly the same, that is 4. The service function chain placement method based on multi-objective deep reinforcement learning according to claim 3, characterized in that, The distributed reinforcement learning includes the formulation of multi-objective Markov decision processes and distributed reinforcement learning structures; Multi-objective Markov decision process: A multi-objective Markov decision process (MOMDP) ​​is represented by tuples as follows: in Representing the state space, Represents the action space. Represents the state transition probability matrix. For a collection of reward functions, Let f represent the weight preference space, and let f be the function that maps policy π to scalar values. 1) State: State It is the collection of physical network information and network service requirements that all intelligent agents can perceive, and the state information X′ encoded by STE. t The input information for the agent is the state information of the agent at time step t. It can be represented as 2) Actions: In the face of continuously and dynamically changing network states, actions are... Defined as discrete; at the current time step t, the decision is made to place the VNF. Physical network infrastructure node n t Let this be denoted as action a. t , Its definition is shown in formula (7); 3) Rewards: It is the set of reward functions for all agents; in the state When using action a t The intelligent system can obtain rewards by interacting with the network environment. The reward value is used to evaluate whether an action is worth choosing; here, the reward value is set to negative, and the reward... It consists of two parts: the first part is the reward for the legality of VNF placement, and the second part is the reward for the consumption incurred during VNF placement; the reward for the legality of placement is denoted as... Before placement, it is necessary to determine the physical network node n. t Remaining resources Does the current VNF ​​request resource satisfy the current request, and (n t-1 ,n t Remaining bandwidth resources of inter-node links Does the SFC's requested resource meet the requirements? If both are met, then the current action a is considered successful. t If it's valid, the reward value is 0; otherwise, the reward value will be... Setting it to a minimum value terminates the current VNF ​​placement action and resets the previous SFC. The VNF already placed in i s,o Release, indicating a request to deny; if action a t If it is valid, then the reward for resource consumption during placement needs to be further calculated, i.e., the consumption of the optimization objective. This part of the reward will vary depending on the agent. Specifically, the agents involved have different optimization objectives. According to the optimization objectives defined above, they are end-to-end latency D. total With network congestion rate U res It will reward value They are and Its definition is shown in formula (8); Reward value obtained by the agent The definition is shown in formula (9); 4) Policy Update: In MOMDP, a policy is a mapping π from a specific state to an action. The agent learns the optimal strategy by interacting with the corresponding environment in order to obtain LTECR; The action value function of strategy π is used This indicates that at time t, the state is... Take action next time t The value of is defined as shown in formula (10), following the expected return of strategy π; Combine weights and experience Together, they are stored in the agent's experience pool, where the Q-network is used to approximate the Q-function of the target k, and the state... The Q-value vector of the optimization objective k is represented as follows: Its definition is shown in formula (11), where Weight their preferences; For the update of the Q function, as defined in formula (12), α is the step size scalar. Let θ be the objective Q function, defined as shown in formula (13), where θ t and θ′ t These represent the network parameter weights of the Q-value function and the network parameter weights of the target Q-value function, respectively. Distributed reinforcement learning architecture: The distributed reinforcement learning structure sets up k agents and considers two optimization objectives: end-to-end latency and network congestion rate. The design of the entire distributed structure has two core elements: Q-value and action remapping, dynamic weight mechanism and diverse experience replay. 1) Remapping of Q-values ​​to actions: The z-score normalization method was used to... Standardization is performed, and its definition is shown in formula (14), where μ represents the Q-value vector. The mean value is σ, and the standard deviation is defined as shown in formulas (15) and (16). 2) Dynamic weighting mechanism and diverse experience replay: Map different service request tasks to SFC placement subproblems i with different weights, and assign them specific weight combinations. The weighted sum of the values ​​in the weighted combination is 1. This represents the user's preference for different optimization objectives k in the current subproblem i. The joint Q-value vector is defined as shown in formula (17): The agent selects actions using an ε-greedy policy, defined as shown in equation (18): Where prob. is the abbreviation for probability, and ε, ε∈[0,1] are the probabilities of random exploration and annealing with time step.

Images