Virtual network function deployment method in 6G network based on deep reinforcement learning

Abstract

The invention provides a virtual network function deployment method in a 6G network based on deep reinforcement learning. The method comprises the following steps: modeling a 6G network function virtualization environment system; a deployment problem of a virtual network function VNF in a 6G network function virtualization environment system is modeled as a Markov decision process MDP, and a state space S, an action space A, a state transition P, a reward function R and a discount factor gamma of the MDP process are defined. And training an intelligent agent based on an offline training algorithm of the DQN, and intelligently solving the MDP process through a deep reinforcement learning algorithm to obtain an optimal VNF deployment position, a flow routing path and an optimal VNF processing and data flow transmission opportunity. According to the invention, the deployment position and resource allocation of the VNF can be dynamically adjusted and controlled according to the network state and the service demand, so that the quality and efficiency of network service are improved. The model can provide quantitative decision support for network service orchestration, and maximizes the resource utilization rate and the network performance while ensuring that the service delay requirement is met.

Description

Technical Field

[0001] The present invention relates to the field of computer application technologies, and particularly to a method for deploying virtual network functions in a 6G network based on deep reinforcement learning. Background Art

[0002] With the advent of the 6G era, we are moving towards a brand-new intelligent era, in which the interconnection of all things and virtual symbiosis will become a reality. The 6G network will greatly improve the digital life quality of the people. However, 6G poses higher requirements for the personalization, adaptability, and resource utilization efficiency of future network service functions. The 6G network is not just an improvement in speed; it also represents a brand-new network service mode. This mode emphasizes the immediacy, reliability, and intelligence of services, especially in the face of emergency situations and critical tasks, such as scenarios like remote medical surgery, disaster response, and autonomous driving. These applications have extremely strict requirements for the latency and stability of the network, and any minor delay may lead to serious consequences.

[0003] With the development of artificial intelligence, the data traffic in the 6G network may experience an explosive growth. How to efficiently utilize limited resources, reduce energy consumption, and at the same time meet the growing service demands has become an urgent problem to be solved. Facing the extensiveness, complexity, and immediate response requirements of latency-sensitive services, the management of the network faces unprecedented challenges. The characteristics of these services require us not only to focus on the deployment and execution of services but also to deeply consider how to achieve refined traffic control and resource scheduling at the network level. This research not only involves technological innovation but also relates to how to meet the high standards of future network services in terms of performance, efficiency, and user satisfaction.

[0004] In order to optimize the deployment cost of virtual deep packet inspection functions in the NFV infrastructure, the prior art has proposed a cost optimization method in intelligent service function chain orchestration. This method aims to minimize the operating cost while meeting the requirements of traffic management and network security, which includes aspects such as licensing fees, network efficiency, and power consumption. The vDPI function deployment problem is defined as a cost minimization problem through mathematical modeling, and an integer linear programming (ILP) model is used for solution. The ILP model aims to select the best deployment location of vDPI, weighing the deployment cost on different network nodes against the overall network resource consumption to achieve efficient utilization of resources.

[0005] The disadvantages of the cost optimization method in intelligent service function chain orchestration in the above prior art include: With the continuous progress of virtualization technology and the emergence of new applications, users' demands for network services have become more diverse and complex. Although this method meets the requirements of traffic management and network security by considering the deployment of vDPI (virtual deep packet inspection) service functions, it often lacks in-depth consideration of path selection. This is particularly insufficient in the current environment because users not only require low-latency service quality but also expect VNFs to be efficiently embedded in the network and routing issues to be properly solved.

[0006] A method for virtual network function (VNF, Virtual Network Function) placement and scheduling for delay-sensitive service configuration optimization in the prior art includes: modeling VNFs as an ILP problem and also proposing two heuristic algorithms to find approximate solutions: a greedy-based algorithm and a Tabu search-based algorithm. The greedy-based algorithm quickly constructs a solution through a series of locally optimal choices and is suitable for application scenarios that require quick decisions; while the Tabu search-based algorithm uses a memory structure to avoid revisiting the same solution, thus jumping out of the local optimum and exploring a wider solution space to increase the likelihood of finding a better or even global optimum solution.

[0007] The disadvantages of the above method for VNF placement and scheduling for delay-sensitive service configuration optimization in the prior art include: The greedy algorithm may encounter causality problems, that is, early choices may limit subsequent choices, thus affecting the quality of the overall solution. Although the Tabu search-based algorithm can avoid being trapped in the local optimum through the memory mechanism and explore a wider solution space, this method requires more computing resources and time to evaluate different solutions. For application scenarios with limited real-time performance and computing resources, this is a significant challenge.

[0008] A heuristic reliability-aware service function chain (SFC, Service Function Chain) embedding method in the prior art includes: This method focuses on services in a network function virtualization (NFV, Network Functions Virtualization) environment, especially for those applications with extremely high requirements for latency and reliability, and provides a new method to optimize the SFC embedding problem in the NFV environment, with particular attention to reducing latency and improving reliability.

[0009] The disadvantages of a heuristic reliability-aware SFC embedding method in the above prior art include: This method mainly studies the offline allocation scenario, that is, in the case of known network topology and SFC requests, optimizing resource allocation and path selection. However, the traffic in the actual network environment has significant dynamics, and the arrival and departure of traffic will change continuously over time, resulting in fluctuations in network resource requirements and allocations. Summary of the Invention

[0010] An embodiment of the present invention provides a virtual network function deployment method in a 6G network based on deep reinforcement learning to maximize the resource utilization rate and network performance of the 6G network.

[0011] To achieve the above object, the present invention adopts the following technical solutions.

[0012] A virtual network function deployment method in a 6G network based on deep reinforcement learning includes:

[0013] Modeling the 6G network function virtualization environment system;

[0014] Model the deployment problem of virtual network functions (VNFs) in the 6G network function virtualization environment system as a Markov decision process (MDP), and define the state space S, action space A, state transition P, reward function R, and discount factor γ of the MDP process;

[0015] Train an agent based on the offline training algorithm of DQN to obtain a trained agent, and use the trained agent to solve the MDP process through the deep reinforcement learning (DRL) algorithm to obtain the optimal VNF deployment location and flow routing path.

[0016] Preferably, the modeling of the 6G network function virtualization environment system includes:

[0017] Divide the time of the VNF scheduling system into |T| discrete times to form a time set T = {1, 2,..., t,..., |T|}, and use a weighted directed graph G = (V, E) to represent the network topology of the 6G network function virtualization environment system, where V is the set of server nodes and E is the set of physical links. Use multiple matrices to represent the availability of network states in different time slots. The available computing resources of node v in time slot t are represented as The available bandwidth of the link from node u to node v is represented as For the initial moment t = 0, the available computing and bandwidth resources are both in the initial state, and there are and

[0018] Modeling of the SFC request collection: The service requests initiated by users are NS = {ns1,..., nsi ,..., ns NS}, where ns i represents a tuple sent by each SFC request, which includes the source address src i and the destination address dst i , the ordered network that constitutes the network service SFC computing resource requirements bandwidth resource requirements the maximum allowed network latency time and the SFC arrival time l i ;

[0019] Define binary VNF mapping variables and virtual link mapping variables as described as the network service ns i ∈ VNFf in NS i k ∈ F i Whether it has been deployed on the node v ∈ V within the time slot t ∈ T, judge whether the available computing resources of the node can meet the VNF resource requirements to be deployed during the current time period. If it can be met, then If it cannot be met, then That is, it is necessary to wait for the node to have enough idle resources to host this VNF or re - select a node capable of hosting this VNF by increasing the delay;

[0020] as described as VNFf i k ∈ F i and f i k' ∈ F i Whether the route between and f i k ∈ F i and its subsequent adjacent VNFf i k' ∈ F i has been mapped on the physical link between the nodes v ∈ V and u ∈ V within the time slot t ∈ T, judge whether the available bandwidth resources of the link can meet the resource requirements between the VNFf i k ∈ F i and its subsequent adjacent VNFf i k' If it can be met, then If it cannot be met, then

[0021]

[0022] The modeling of node resource utilization is:

[0023] Among them, takes values of 0 or 1, indicating whether VNFf i k is embedded into the time slot t ∈ T of node v ∈ V, represents the initial available computing resources of node v ∈ V, represents when VNFf i k is deployed on node v ∈ V, the computing resources consumed;

[0024] The modeling of link resource utilization rate is:

[0025]

[0026] Among them, takes values of 0 or 1, indicating whether VNFf i k and VNFf i k' the virtual link between them is mapped to the link (v, u) ∈ E in time slot t ∈ T, represents the initial available bandwidth resources of the physical link between node v ∈ V and node u ∈ V, represents the network service ns i ∈ NS's demand for bandwidth resources;

[0027] The modeling of service delay guarantee degree is:

[0028]

[0029] Among them, β represents the delay guarantee weight coefficient of the service;

[0030] Node resource constraint is to ensure that in any time slot, the total resource demand of all VNF instances deployed on the node does not exceed the maximum available computing resources of the node;

[0031]

[0032] Among them, represents whether VNFf i ∈ NS in the network service ns i k ∈ F i has been deployed on node v ∈ V within time slot t ∈ T, represents when VNFf i k is deployed on node v ∈ V, the computing resources consumed, represents the available computing resources of node v ∈ V at time slot t ∈ T;

[0033] Link resource constraints ensure that within any time slot, the total bandwidth demand of all VNF pairs mapped to the link does not exceed the maximum available bandwidth resource of the link;

[0034]

[0035] Among them, represents whether the route between VNFf i k ∈F i and f i k' ∈F i has been mapped to the physical link between nodes v∈V and u∈V at time slot t∈T, represents the bandwidth resource demand of network service ns i ∈NS, represents the available bandwidth of the link from node v∈V to node u∈V at time slot t∈T;

[0036] There is a time gap between changes in node and link resources in the same network service. The network conditions for VNFs deployed earlier should be greater than those for VNFs deployed later;

[0037]

[0038]

[0039] Among them, represents the available computing resource of node v∈V at time slot t∈T, represents the available computing resource of node v∈V at time slot t'∈T, represents the available bandwidth of the link from node v∈V to node u∈V at time slot t∈T, represents the available bandwidth of the link from node v∈V to node u∈V at time slot t'∈T. Correspondingly, time slot t'∈T occurs after time slot t∈T;

[0040] Ensure that the end-to-end delay L of the SFC i does not exceed the maximum acceptable delay of the service request The delay constraint is:

[0041]

[0042] Among them, L prop ,L proc ,L queue represent propagation delay, processing delay, and queuing delay respectively;

[0043] Each VNF requested by the SFC will not be repeatedly mapped to multiple network nodes or multiple physical links throughout the system time.

[0044]

[0045]

[0046] Among them, represents whether the VNF f i ∈F in the network service ns i k ∈F i has been deployed on the node v ∈ V within the time slot t ∈ T, represents whether the routing between the VNF f i k ∈F i and f i k' ∈F i has been mapped to the physical link between the nodes v ∈ V and u ∈ V within the time slot t ∈ T;

[0047] Subsequent VNF processing must be carried out after the previous VNF processing is completed.

[0048]

[0049] Among them, represents whether the VNF f i ∈F in the network service ns i k ∈F i has been deployed on the node v ∈ V within the time slot t ∈ T, represents whether the VNF f i ∈F in the network service ns i k' ∈F i has been deployed on the node u ∈ V within the time slot t' ∈ T;

[0050] The total resource requirement of the SFC cannot exceed the maximum available computing resource of the node, and the total allocated bandwidth requirement cannot exceed the maximum available bandwidth of the link.

[0051]

[0052]

[0053] Among them, represents whether the VNF f i ∈F in the network service ns i k ∈F iWhether it has been deployed on node v∈V within time slot t∈T, Indicates the computing resources consumed when VNFf i k is deployed on node v∈V, Indicates the initial available computing resources of node v∈V, Indicates VNFf i k ∈F i and f i k' ∈F i Whether the route between and f Indicates the bandwidth resource requirement of network service ns i ∈NS, Indicates the initial available bandwidth resources of the physical link between node v∈V and node u∈V.

[0054] Preferably, the deployment problem of VNF in the 6G network function virtualization environment system is modeled as a Markov decision process MDP, and the state space S, action space A, state transition P, reward function R, and discount factor γ of the MDP process are defined, including:

[0055] Model the deployment problem of VNF as a Markov decision MDP process, define the state space S, action space A, state transition P, and reward function R of the MDP process, and the discount factor γ is used to weigh the importance of future rewards, expressed as a quadruple M=(S,A,P,R,γ);

[0056] State space S: Includes the underlying network state S net and service request state S req , where S net Includes the available computing resources on |V|×1×|T| The available bandwidth resources on |V|×|V|×|T| And the delay situation of the links on |V|×|V|×|T| While S req Includes the node resource requirements of |F|×|V|, the link resource requirements on |F|×1, and the maximum allowable delay;

[0057] Action space A: Determine the range of actions that the agent can take at each decision moment, and define the VNF deployment location and embedding time

[0058] State transition P: Includes the current state s of the agent n and the next state s after executing action a n n ​', which describes the probability distribution of the system transitioning to the next state given the current state and the actions taken;

[0059] Discount factor γ: used to measure the importance of future rewards for current decisions;

[0060] The expression of the reward function R is as follows:

[0061]

[0062] where ψ c and ψ b and φ l respectively represent the node resource utilization rate, the link resource utilization rate, and the service delay guarantee degree.

[0063] Preferably, the DQN-based offline training algorithm is used to train the agent to obtain a trained agent, and the DRL algorithm is used to utilize the trained agent to solve the MDP process to obtain the optimal VNF deployment location and flow routing path, including:

[0064] The DQN-based offline training algorithm is used to train the agent to obtain a trained agent. The DRL algorithm is used to utilize the trained agent to solve the MDP process. The trained agent makes an optimal decision in the current network state, selects the optimal action, and determines the optimal VNF deployment location and flow routing path according to the optimal action.

[0065] Preferably, the DQN-based offline training algorithm is used to train the agent to obtain a trained agent, including:

[0066] Step 1: Define the state space, action space, and reward function, and construct the MDP process;

[0067] Step 2: Initialize the parameters θ and θ of the main network and the target network - , and set the maximum number of training rounds;

[0068] Step 3: Before training starts, initialize the environmental state;

[0069] Step 4: For each system state ns i and its corresponding traffic f i k , select an action according to the ε-greedy algorithm If the computing resource constraint is satisfied, execute the selected action and calculate the sub-routing strategy. If the maximum tolerable delay and link resource constraint are further satisfied, allocate the corresponding computing and network resources; otherwise, terminate the current service request and release the allocated resources;

[0070] Step 5: After executing the action, r nand enter the next state s n ', store the quadruple (s n , a n , r n , s n ) of the current state, action, reward, and next state into the experience replay buffer;

[0071] Step 6: When the number of samples in the experience replay buffer exceeds a certain threshold, randomly draw a small batch of data from the buffer for discrete training, use these small batches of data to calculate the target network, and update the parameters θ of the current network accordingly, and update the parameters θ of the target network - to keep it synchronized with the current network;

[0072] Step 7: Repeat Steps 3 - 6 until the preset maximum number of training rounds is reached to obtain a trained agent.

[0073] As can be seen from the technical solutions provided by the present invention above, the present invention establishes a service chain optimization solution that meets the network function virtualization (NFV) and software-defined network (SDN) environments. Through a flexible virtual network function (VNF) deployment solution, it can cope with the diverse and dynamically changing demand scenarios that appear in future networks. This solution can dynamically control the deployment location, routing path, VNF processing, and traffic transmission timing of VNFs according to the network state and service requirements to improve the quality and efficiency of network services. This model can provide a quantitative decision-making support for network service orchestration to ensure maximizing resource utilization and network performance while meeting service latency requirements.

[0074] Additional aspects and advantages of the present invention will be given in part in the following description, which will become apparent from the following description or be understood through the practice of the present invention. Description of the Drawings

[0075] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0076] Figure 1 It is a processing flow chart of a VNF deployment method in a 6G network based on deep reinforcement learning provided by an embodiment of the present invention. Detailed Embodiments

[0077] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where like or similar reference numerals denote like or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0078] Those skilled in the art of the present technology can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The phrase "and / or" used herein includes any and all combinations of any one of the one or more associated listed items.

[0079] Those skilled in the art of the present technology can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the field to which the present invention belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art and will not be interpreted with an idealized or overly formal meaning unless defined as herein.

[0080] For the convenience of understanding the embodiments of the present invention, the following will further explain with several specific embodiments as examples in conjunction with the accompanying drawings, and each embodiment does not constitute a limitation to the embodiments of the present invention.

[0081] The embodiments of the present invention design a VNF deployment mechanism based on deep reinforcement learning (DRL, Deep Reinforcement Learning), combine resource and delay optimization strategies, and introduce a Markov decision process (MDP, Markov Decision Process) model. By continuously interacting with the environment using the DRL algorithm, an optimal strategy for resource utilization and delay guarantee is obtained, reducing the impact of network latency and service rejection. This mechanism can adaptively adjust the deployment and traffic routing of VNFs, and at the same time control the timing of VNF processing and traffic transmission to meet the requirements of delay-sensitive services.

[0082] For the dynamicization of network resources and the flexible deployment of VNFs, the model construction was continued and supplemented. Through the description of the ILP problem, the Quality of Service (QoS) was taken as the optimization goal, and the spatio-temporal dynamics of node computing resources and link bandwidth resources were considered simultaneously.

[0083] To ensure the normal and efficient operation of SFC in delay-sensitive scenarios, the present invention designs an optimization and orchestration method for service function chains with joint resource and delay awareness in a 6G network function virtualization environment based on deep reinforcement learning. Through the cooperation of the data plane, control plane, and knowledge plane, by real-time monitoring of network states (including the computing resources of nodes, the bandwidth resources of links, and network delays, etc.), dynamic adjustment of resource allocation is achieved to adapt to the changing network demands and quality of service requirements.

[0084] The architecture of a virtual network function deployment method in a 6G network based on deep reinforcement learning provided by an embodiment of the present invention is divided into three main parts: the data plane, the control plane, and the knowledge plane. The data plane is mainly responsible for the data forwarding function of the network. Its main task is to ensure that data packets can be correctly forwarded according to the rules issued by the control plane, including switches responsible for data packet forwarding and servers capable of hosting VNFs. The knowledge plane is the intelligent decision-making center in the network architecture, responsible for parsing user service requests and constructing deployment strategies based on dynamic resource changes and underlying network information. The control plane is responsible for monitoring and managing network states, such as delays, bandwidth, etc., and collecting computing and network resource information. The control plane includes a controller, an orchestrator, and a collector. The orchestrator is responsible for managing and coordinating resources and services in the network to ensure QoS and Quality of Experience (QoE). The control plane constructs deployment strategies according to dynamic resource changes, such as demands, service delays, and other indicators, as well as underlying network information. The coordinator will guide the deployment of VNFs according to the strategies provided by the knowledge plane, and the controller sends traffic forwarding rules to the underlying network to control the direction of traffic. In short, these three planes cooperate with each other to jointly achieve the efficient management and optimization of network services.

[0085] The working process of the above architecture includes:

[0086] The user initiates a service request according to their own needs, and this request may include specific quality of service requirements, delay limitations, and other service parameters.

[0087] The knowledge plane receives the user's service request, simultaneously parses the service request, and uses a coordination mechanism to analyze the required network resources, computing resources, and the delay sensitivity of the service.

[0088] Based on the analysis results of service requests, the knowledge plane formulates corresponding deployment strategies and issues the formulated deployment strategies and decisions to the coordinator in the control plane in the form of instructions.

[0089] The coordinator converts the policies and decisions into JSON - formatted policies to guide the deployment of VNFs.

[0090] The control plane will monitor the network status in real - time, including the computing resources of nodes, the bandwidth resources of links, and network latency, etc. At the same time, the control plane will dynamically allocate and adjust computing resources and network resources according to the policies issued by the knowledge plane to meet service requests.

[0091] The control plane issues specific network configurations and behavior policies to the data plane to guide the data forwarding behavior of the data plane.

[0092] The data plane receives, processes, and forwards data packets according to the rules issued by the control plane. Or execute VNFs on the server, such as firewalls, load balancers, etc.

[0093] The data plane feeds back the network status to the control plane in real - time for network status monitoring and resource scheduling. At the same time, the control plane adjusts the real - time resource management policy according to the monitoring results fed back by the data plane.

[0094] The control plane feeds back the adjustment results to the knowledge plane so that the knowledge plane can perform further policy optimization.

[0095] The knowledge plane intelligently optimizes and adjusts the service deployment policy according to the feedback and learning results, forming a closed - loop optimization process.

[0096] The processing flow of a VNF deployment method for a 6G network based on deep reinforcement learning provided by an embodiment of the present invention is as Figure 1 shown, including the following processing steps:

[0097] Step S10: Model the 6G network function virtualization environment system.

[0098] Modeling of the network topology of a 6G network function virtualization environment system provided by an embodiment of the present invention: Divide the time of the VNF scheduling system into |T| discrete time periods, forming a time set T={1,2,...,t,...,|T|}. Use a weighted directed graph G=(V,E) to represent the network topology, where V is the set of server nodes and E is the set of physical links. Use multiple matrices to represent the availability of network states in different time slots. The available computing resources of node v in time slot t are represented as The available bandwidth of the link from node u to node v is represented as For the initial moment t = 0, the available computing and bandwidth resources are both in the initial state, and there is and

[0099] Modeling of Service Function Chain Request Aggregates: The service requests initiated by users are NS = {ns1,..., ns i ,..., ns NS}, where ns i represents a tuple of each SFC request, which includes the source address src i and the destination address dst i , the ordered network that constitutes the network service SFC computing resource requirements bandwidth resource requirements the maximum allowed network latency time SFC arrival time l i .

[0100] Define binary VNF mapping variables and virtual link mapping variables Specifically, is described as whether the VNF f i ∈ NS in the network service ns i k ∈ F i has been deployed on the node v ∈ V within the time slot t ∈ T. Determine whether the available computing resources of the node can meet the resource requirements of the VNF to be deployed during the current time period. If it can be met, it is If it cannot be met, it is That is, it is necessary to wait for the node to have sufficient idle resources to host this VNF or reselect a node capable of hosting this VNF by increasing the delay. is described as whether the route between VNF f i k ∈ F i and f i k' ∈ F i has been mapped on the physical link between the nodes v ∈ V and u ∈ V within the time slot t ∈ T. Determine whether the available bandwidth resources of the link can meet the resource requirements between the VNF f i k ∈ F i and its subsequent adjacent VNF f i k' ∈ F i . If it can be met, it is If it cannot be met, it is It means waiting for the link to have transmission capacity for transmission, or selecting to map to other links.

[0101] The optimization objective of the present invention is to provide maximum QoS guarantee for 6G service function chains, which can be measured by the node resource utilization rate ψ c , the link resource utilization rate ψ b and the service delay guarantee degree φ l .

[0102] Modeling of node resource utilization rate:

[0103]

[0104] Among them, takes values of 0 or 1, indicating whether VNFf i k is embedded in the time slot t∈T of the node v∈V. represents the initial available computing resource of the node v∈V, represents the computing resource consumed when VNFf i k is deployed on the node v∈V.

[0105] Modeling of link resource utilization rate:

[0106]

[0107] Among them, takes values of 0 or 1, indicating whether the virtual link between VNFf i k and VNFf i k' is mapped to the link (u,v)∈E in the time slot t∈T. represents the initial available bandwidth resource of the physical link between the node v∈V and the node u∈V, represents the demand for bandwidth resources of the network service ns i ∈NS.

[0108] Modeling of service delay guarantee degree:

[0109]

[0110] Among them, β represents the delay guarantee weight coefficient of the service.

[0111] The node resource constraint ensures that in any time slot, the total resource demand of all VNF instances deployed on the node does not exceed the maximum available computing resource of the node.

[0112]

[0113] Among them, represents VNFf in the network service ns i ∈NS ik ∈F i Whether it has been deployed on node v ∈ V within time slot t ∈ T. Indicates the computing resources consumed when VNF f i k is deployed on node v ∈ V. Indicates the available computing resources of node v ∈ V at time slot t ∈ T.

[0114] Link resource constraint ensures that within any time slot, the total bandwidth demand of all VNF pairs mapped to a link does not exceed the maximum available bandwidth resource of the link.

[0115]

[0116] Among them, Indicates VNF f i k ∈F i and f i k' ∈F i Whether the route between and f has been mapped on the physical link between node v ∈ V and u ∈ V within time slot t ∈ T. Indicates the bandwidth resource demand of network service ns i ∈ NS. Indicates the available bandwidth of the link from node v ∈ V to node u ∈ V at time slot t ∈ T.

[0117] There is a time gap between changes in node and link resources in the same network service. Meanwhile, the network conditions for VNFs deployed earlier should be greater than those for VNFs deployed later.

[0118]

[0119]

[0120] Among them, Indicates the available computing resources of node v ∈ V at time slot t ∈ T, Indicates the available computing resources of node v ∈ V at time slot t' ∈ T. Indicates the available bandwidth of the link from node v ∈ V to node u ∈ V at time slot t ∈ T, Indicates the available bandwidth of the link from node v ∈ V to node u ∈ V at time slot t' ∈ T. Accordingly, time slot t' ∈ T occurs after time slot t ∈ T.

[0121] Delay constraint ensures that the end-to-end delay L of the service function chain (SFC) i does not exceed the maximum acceptable delay of the service request

[0122]

[0123] Among them, L prop , L proc , L queue respectively represent the propagation delay, processing delay, and queuing delay.

[0124] Each virtualized network function (VNF) requested by the service function chain (SFC) will not be repeatedly mapped to multiple network nodes within the entire system time, nor will it be repeatedly mapped to multiple physical links.

[0125]

[0126]

[0127] Among them, represents whether the VNFf i ∈NS in the network service ns i k ∈F i has been deployed on the node v∈V within the time slot t∈T, represents whether the route between the VNFf i k ∈F i and f i k' ∈F i has been mapped on the physical link between the nodes v∈V and u∈V within the time slot t∈T.

[0128] Subsequent virtualized network function (VNF) processing must be carried out after the previous virtualized network function (VNF) processing ends.

[0129]

[0130] Among them, represents whether the VNFf i ∈NS in the network service ns i k ∈F i has been deployed on the node v∈V within the time slot t∈T, represents whether the VNFf i ∈NS in the network service ns i k' ∈F i has been deployed on the node u∈V within the time slot t'∈T.

[0131] The discrete-time resource awareness constraints include node computing resource constraints and link bandwidth resource constraints, that is, it is required that the total resource demand of the service function chain (SFC) does not exceed the maximum available computing resources of the node, and at the same time, the total allocated bandwidth demand does not exceed the maximum available bandwidth of the link.

[0132]

[0133]

[0134] Among them, represents whether the VNFf i ∈NS in the network service ns i k ∈F i has been deployed on the node v∈V within the time slot t∈T, represents the computing resources consumed when the VNFf i k is deployed on the node v∈V, represents the initial available computing resources of the node v∈V. represents whether the route between the VNFf i k ∈F i and f i k' ∈F i has been mapped to the physical link between the nodes v∈V and u∈V within the time slot t∈T, represents the bandwidth resource demand of the network service ns i ∈NS, represents the initial available bandwidth resources of the physical link between the nodes v∈V and u∈V.

[0135] Step S20: Model the deployment problem of VNFs in the 6G network function virtualization environment system as a Markov Decision Process (MDP), and define the state space S, action space A, state transition P, reward function R, and discount factor γ of the MDP process.

[0136] Based on the above mathematical model, to achieve efficient resource utilization and strict control of latency to meet the requirements of 6G latency-sensitive services such as virtual reality, remote medicine, and autonomous driving. Model the deployment problem of VNFs as a Markov Decision Process (MDP), define the state space S, action space A, state transition P, and reward function R of the MDP process, and the discount factor γ is used to weigh the importance of future rewards, expressed as a quadruple M=(S,A,P,R,γ).

[0137] The specific meanings of each parameter are as follows:

[0138] State space S: includes the underlying network state S net and service request state S req , where S net includes the available computing resources on |V|×1×|T| the available bandwidth resources on |V|×|V|×|T| and the latency conditions of the links on |V|×|V|×|T| while S req includes the node resource requirements of |F|×|V|, the link resource requirements on |F|×1, and the maximum allowable latency.

[0139] Action space A: defines the range of actions that the agent can take at each decision-making moment. By defining the VNF deployment location and embedding time, the agent can select appropriate actions in the action space according to the information in the state space (such as node resource availability, link bandwidth, etc.), so as to achieve reasonable deployment and routing planning of VNFs.

[0140] State transition P: includes the current state s of the agent n and the next state s n ' after executing the action a. It describes the probability distribution of the system transitioning to the next state given the current state and the action taken. n ' after executing the action a. It describes the probability distribution of the system transitioning to the next state given the current state and the action taken.

[0141] Discount factor γ: used to measure the importance of future rewards for the current decision. When calculating the long-term cumulative reward, future rewards will be discounted according to this exponent.

[0142] Reward function R: The goal of the present invention is to bring a smoother network experience to 6G users, that is, to improve QoS. Therefore, the setting of the reward function closely revolves around the QoS functions that the system can provide, aiming to encourage the agent to comprehensively consider multiple factors when making VNF deployment and routing decisions to achieve the optimization of overall QoS. It reflects the performance of the system under different decisions and provides the agent with a clear feedback signal, enabling it to learn the optimal strategy. The specific expression of the reward function R is as follows:

[0143]

[0144] where ψ c , ψ b , φ l respectively represent the node resource utilization rate, the link resource utilization rate, and the service latency guarantee degree.

[0145] Step S30: Train the agent using the offline training algorithm based on DQN (Deep Q-Network) to obtain a trained agent. Use the DRL algorithm to utilize the trained agent to solve the MDP process, and obtain the optimal VNF deployment location and flow routing path.

[0146] With this method, the 6G network intelligent service function chain orchestration process is regarded as a sequential decision-making process, enabling the algorithm to make decisions based on the current state. Train the agent offline to obtain a trained agent. Then, use the trained agent to select the most appropriate action and make an optimal decision in the current network state, that is, select the optimal VNF deployment location and flow routing path, thereby maximizing the long-term cumulative reward and effectively improving the overall performance of 6G network services.

[0147] Specifically, in each round of offline training, the agent observes the current system state s n and interacts with it continuously. According to the ε-greedy strategy based on exploration and exploitation, randomly select an action with a probability of (1 - ε), and select the corresponding action when the Q function is the largest with a probability of ε. Through the ε-greedy strategy, the agent continuously learns unknown state-action pairs to explore the unknown environment. Then, by executing the action a n in the current system state and obtaining the corresponding system reward r n , the system state s n transfers to the next state s n '. At the same time, store this generated set of state transition samples (s n , a n , r n , s n ) in the experience replay pool. When the number of stored samples reaches a certain amount, perform the discrete training process by randomly sampling a small batch of samples from the experience replay pool.

[0148] In offline training, the agent interacts with the environment, selects actions according to the ε-greedy strategy, obtains rewards after executing actions and enters the next state, and at the same time stores the state transition samples in the experience replay pool to provide rich and reliable data support for subsequent network parameter updates. By continuously repeating this process, the agent gradually optimizes its decision-making strategy through a large number of attempts and learning, continuously improves its performance in the 6G network intelligent service function chain orchestration task, and lays a solid algorithm foundation for realizing efficient and high-quality 6G network services. The specific algorithm is as follows

[0149]

[0150]

[0151] The specific process of the algorithm includes the following steps:

[0152] (1) Step 1: Define the state space, action space, and reward function, and construct a Markov decision process (MDP)

[0153] (2) Step 2: Initialize the parameters θ and θ of the main network and the target network - . At the same time, set the maximum number of training rounds;

[0154] (3) Step 3: Before starting the training, initialize the environmental state;

[0155] (4) Step 4: For each system state ns i and its corresponding traffic f i k , select an action according to the ε-greedy algorithm If the computing resource constraint is satisfied, execute the selected action and calculate the sub-routing strategy. If the maximum tolerable delay and link resource constraints are further satisfied, allocate the corresponding computing and network resources; otherwise, terminate the current service request and release the allocated resources;

[0156] (5) Step 5: After executing the action, r n and enter the next state s n '. At the same time, store the quadruple (s n , a n , r n , s n ') of the current state, action, reward, and next state into the experience replay buffer pool;

[0157] (6) Step 6: When the number of samples in the experience replay buffer pool exceeds a certain threshold, randomly draw a small batch of data from the buffer pool for discrete training. Use these small batches of data to calculate the target network and update the parameters θ of the current network accordingly. Then, update the parameters θ of the target network - to keep it synchronized with the current network.

[0158] (7) Step 7: Repeat steps 3 - 6 until the preset maximum number of training rounds is reached to obtain a trained agent.

[0159] The goal of this algorithm is to learn how to make optimal decisions in different network states by interacting with the environment repeatedly, that is, to select the optimal VNF deployment location and flow routing path, so as to maximize the quality of service (QoS) and ensure low latency and efficient resource utilization. In addition, through offline training, the SFC scheduling strategy can be optimized without disturbing the real-time network operation.

[0160] Driven by 6G network technology, key application areas sensitive to latency, such as intelligent driving, Internet of Things (IoT), and telemedicine, will witness significant development and widespread application. The Orchestrator plays a crucial role in the 6G network.

[0161] In summary, based on this, the present invention combines the MDP model with the deep reinforcement learning algorithm and innovatively proposes a joint network resource allocation and delay-aware service function chain optimization and orchestration algorithm. By establishing a refined network resource model, this algorithm can adjust and optimize the deployment and transmission path of network services in real time in a dynamically changing network environment to meet the requirements of 6G networks for high efficiency, low latency, and resource optimization. The network architecture design of the present invention is ingenious and aims to achieve efficient, flexible, and intelligent network service management.

[0162] The IntelliChain-6G embodiment of the present invention deeply analyzes the complex dependencies among path selection, VNF placement, and routing selection, ensuring that each service request can obtain an optimal resource allocation and traffic routing strategy. This not only improves the utilization efficiency of network resources but also enhances the flexibility of service deployment. At the same time, the introduction of time slot scheduling can ensure that critical tasks are processed during the idle periods in the network, thereby reducing latency. This method is particularly suitable for latency-sensitive application scenarios such as telemedicine surgeries and autonomous driving. By leveraging the DRL algorithm, IntelliChain-6G can automatically learn and obtain an optimal strategy for resource utilization and delay guarantee through continuous interaction with the environment, improving the system's adaptability and intelligence level. In addition, the invention can dynamically adjust the deployment location and resource allocation of VNFs according to changes in real-time network status and service requirements. This flexibility significantly improves the quality and efficiency of network services, especially in the face of rapidly changing network conditions and service requests.

[0163] Those of ordinary skill in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or processes in the drawings are not necessarily essential for implementing the present invention.

[0164] From the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0165] Each embodiment in this specification is described in a progressive manner. For the identical or similar parts among the embodiments, reference can be made to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0166] As mentioned above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for deploying virtual network functions in a 6G network based on deep reinforcement learning, characterized in that: include: Modeling of 6G network function virtualization environment system; The deployment problem of virtual network functions (VNFs) in the 6G network function virtualization environment system is modeled as a Markov decision process (MDP). The state space S, action space A, state transition P, reward function R, and discount factor γ of the MDP process are defined. The intelligent agent is trained based on the offline training algorithm of DQN to obtain a trained intelligent agent, and the trained intelligent agent is used by the deep reinforcement learning DRL algorithm to intelligently solve the MDP process to obtain the optimal VNF deployment location and flow routing path.

2. The method according to claim 1, characterized in that The modeling of the 6G network function virtualization environment system includes: The time of the VNF scheduling system is divided into |T| discrete times to form a time set T = {1, 2, ..., t, ..., |T|}. A weighted directed graph G = (V, E) is used to represent the network topology of the 6G network function virtualization environment system, where V is the set of server nodes and E is the set of physical links. Multiple matrices are used to represent the availability of network status in different time slots. The available computing resources of node v in time slot t are expressed as The available bandwidth of the link from node u to node v is expressed as For the initial time t = 0, the available computing and bandwidth resources are in the initial state, and Modeling of SFC request set: The service request initiated by the user is NS = {ns1,...,ns i ,...,ns |NS| }, where ns i Represents a tuple sent by each SFC request, which includes the source address src of the SFC i and the destination address dst i , forming an orderly network of network services SFC computing resource requirements Bandwidth resource requirements Maximum allowed network delay and SFC arrival time l i ; Define binary VNF mapping variables and virtual link mapping variables Will Described as network service ns i VNFf in ∈NS i k ∈F i Whether it has been deployed on the node v∈V in the time slot t∈T, and whether the available computing resources of the node can meet the resource requirements of the VNF to be deployed in the current time period. If so, If it cannot be satisfied, That is, it is necessary to wait for the node to have enough free resources to host this VNF or reselect a node capable of hosting this VNF by increasing the delay; Will Described as VNFf i k ∈F i and f i k' ∈F i Whether the route between them has been mapped on the physical link between nodes v∈V and u∈V in time slot t∈T, and whether the available bandwidth resources of the link can meet the needs of VNFf to be deployed in the current time period i k ∈F i Its adjacent VNFf i k' ∈F i If the resource requirements between If it cannot be satisfied, Indicates that transmission is performed while waiting for the link to have transmission capability, or the link is mapped to another link. The node resource utilization is modeled as: in, The value is 0 or 1, indicating that VNFf i k Whether it is embedded in the time slot t∈T of node v∈V, represents the initial available computing resources of node v∈V, Indicates that when VNFf i k The computing resources consumed when deployed on node v∈V; The link resource utilization is modeled as: in, The value is 0 or 1, indicating that VNFf i k and VNFf i k' Whether the virtual link between them is mapped to the link (v,u)∈E in the time slot t∈T, represents the initial available bandwidth resource of the physical link between node v∈V and node u∈V, Represents network service ns i ∈NS demand for bandwidth resources; The service delay guarantee is modeled as: Among them, β represents the delay guarantee weight coefficient of the service; The node resource constraint is to ensure that in any time slot, the total resource demand of all VNF instances deployed on the node does not exceed the maximum available computing resources of the node; in, Represents network service ns i VNFf in ∈NS i k ∈F i Whether it has been deployed on node v∈V within time slot t∈T, Indicates that when VNFf i k The computing resources consumed when deployed on node v∈V, represents the available computing resources of node v∈V at time slot t∈T; Link resource constraints ensure that in any time slot, the total bandwidth demand of all VNF pairs mapped to the link does not exceed the maximum available bandwidth resources of the link; in, VNFf i k ∈F i and f i k' ∈F i Whether the route between has been mapped on the physical link between nodes v∈V and u∈V in time slot t∈T, Represents network service ns i ∈NS demand for bandwidth resources, represents the available bandwidth of the link from node v∈V to node u∈V at time slot t∈T; There is a time gap between changes in node and link resources in the same network service. The network conditions for earlier deployed VNFs should be greater than those for later deployed VNFs. in, represents the available computing resources of node v∈V at time slot t∈T, represents the available computing resources of node v∈V at time slot t'∈T, represents the available bandwidth of the link from node v∈V to node u∈V at time slot t∈T, represents the available bandwidth of the link from node v∈V to node u∈V at time slot t′∈T. Accordingly, time slot t′∈T occurs after time slot t∈T; Ensure that the end-to-end delay L of SFC i Do not exceed the maximum acceptable delay in servicing requests The delay constraint is: Among them, L prop , L proc , L queue They are represented as propagation delay, processing delay and queuing delay respectively; Each VNF requested by the SFC will not be repeatedly mapped to multiple network nodes or multiple physical links during the entire system time; in, Represents network service ns i VNFf in ∈NS i k ∈F i Whether it has been deployed on node v∈V within time slot t∈T, VNFf i k ∈F i and f i k' ∈F i Whether the route between has been mapped on the physical link between nodes v∈V and u∈V in time slot t∈T; The subsequent VNF ​​processing must be performed after the previous VNF processing is completed; in, Represents network service ns i VNFf in ∈NS i k ∈F i Whether it has been deployed on node v∈V within time slot t∈T, Represents network service ns i VNFf in ∈NS i k' ∈F i Whether it has been deployed on node u∈V within time slot t'∈T; The total resource requirement of the SFC cannot exceed the maximum available computing resource of the node, and the total allocated bandwidth requirement cannot exceed the maximum available bandwidth of the link; in, Represents network service ns i VNFf in ∈NS i k ∈F i Whether it has been deployed on node v∈V within time slot t∈T, Indicates that when VNFf i k The computing resources consumed when deployed on node v∈V, represents the initial available computing resources of node v∈V, VNFf i k ∈F i and f i k' ∈F i Whether the route between has been mapped on the physical link between nodes v∈V and u∈V in time slot t∈T, Represents network service ns i ∈NS demand for bandwidth resources, Represents the initial available bandwidth resources of the physical link between node v∈V and node u∈V.

3. The method according to claim 2, characterized in that The deployment problem of VNF in the 6G network function virtualization environment system is modeled as a Markov decision process MDP, and the state space S, action space A, state transition P, reward function R and discount factor γ of the MDP process are defined, including: The VNF deployment problem is modeled as a Markov decision MDP process. The state space S, action space A, state transition P, and reward function R of the MDP process are defined, and the discount factor γ is used to weigh the importance of future rewards, which is expressed as a four-tuple M = (S, A, P, R, γ). State space S: includes the underlying network state S net and service request status S req , where S net Including the available computing resources on |V|×1×|T| Available bandwidth resources on |V|×|V|×|T| And the link delay of |V|×|V|×|T| And S req Including the node resource requirements of |F|×|V|, the link resource requirements on |F|×1, and the maximum allowed delay; Action space A: Determines the range of actions that the agent can take at each decision moment, and defines the VNF deployment location and embedding time State transition P: includes the current state s of the agent n and perform action a n The next state after n ', describes the probability distribution of the system moving to the next state given the current state and the action taken; Discount factor γ: used to measure the importance of future rewards to current decisions; The expression of the reward function R is as follows: Among them, ψ c , ψ b ,φ l They represent node resource utilization, link resource utilization, and service delay guarantee respectively.

4. The method according to claim 3, characterized in that The DQN-based offline training algorithm trains the intelligent agent to obtain a trained intelligent agent, and uses the trained intelligent agent to intelligently solve the MDP process through the DRL algorithm to obtain the optimal VNF deployment position and flow routing path, including: The intelligent agent is trained based on the offline training algorithm of DQN to obtain a trained intelligent agent, and the trained intelligent agent is used to solve the MDP process through the DRL algorithm. The trained intelligent agent makes the best decision under the current network state and selects the best action. The optimal VNF deployment location and flow routing path are determined according to the optimal action.

5. The method according to claim 4, characterized in that The DQN-based offline training algorithm trains the intelligent agent to obtain a trained intelligent agent, including: Step 1: Define the state space, action space, reward function, and construct the MDP process; Step 2: Initialize the parameters θ and θ of the main network and the target network - , set the maximum number of training rounds; Step 3: Initialize the environment state before training begins; Step 4: For each system state ns i and its corresponding flow rate f i k , select actions according to the ε-greedy algorithm If the computing resource constraints are met, the selected action is executed and the sub-routing strategy is calculated. If the maximum tolerable delay and link resource constraints are further met, the corresponding computing and network resources are allocated; otherwise, the current service request is terminated and the allocated resources are released; Step 5: After executing the action, r n And enter the next state s n ', the current state, action, reward and next state quadruple (s n ,a n ,r n ,s n ') Stored in the experience replay buffer pool; Step 6: When the number of samples in the experience replay buffer exceeds a certain threshold, randomly extract small batches of data from the buffer for discrete training, use these small batches of data to calculate the target network, and update the parameters θ of the current network and the parameters θ of the target network accordingly. - To keep in sync with the current network; Step 7: Repeat steps 3 to 6 until the preset maximum number of training rounds is reached to obtain a trained agent.

Images