A method and apparatus for digital twin network slicing based on reinforcement learning

Abstract

The application relates to the field of network communication, in particular to a method and device for digital twin network slicing based on reinforcement learning. Mainly comprising: a reinforcement learning module acquires current network information triples and network slicing demand triples in a knowledge graph; the reinforcement learning module generates at least one decision action of network slicing, each decision action contains PNF node searching according to a node selection probability model and the knowledge graph and resource allocation vector mapping of resource allocation to each PNF node according to a resource allocation model, the knowledge graph makes a decision according to the decision action to obtain a decision result, the decision result is a deployment position of network slicing VNF and network slicing PNF; the knowledge graph is updated according to the decision result, and the updated network state is fed back to the reinforcement learning module, and the reinforcement learning module issues a deployment strategy of the network slicing. The application can quickly obtain an optimal slicing deployment scheme in a current network scene under a dynamic network environment.

Description

Technical Field

[0001] This invention relates to the field of network communication, and in particular to a method and apparatus for digital twin network slicing based on reinforcement learning. Background Technology

[0002] With the development of 5G and cloud technologies, more and more new types of services are emerging, and different services or users have different requirements for network service quality. To meet the diverse needs of users and provide personalized services, network slicing technology has emerged. For example... Figure 1 As shown, network slicing technology can flexibly allocate resources on a shared physical network to build multiple dedicated, isolated virtualized logical networks (i.e., multiple network slices), each designed to meet specific business needs. Network slicing is implemented through network virtualization technology and consists of a set of Virtual Network Functions (VNFs) that meet business requirements, along with links connecting each pair of VNFs. Network slicing can be viewed as a process of allocating resources for VNFs and links based on a shared network infrastructure. How to flexibly and efficiently allocate limited resources to meet the service needs of each network slice is a hot research topic in network slicing.

[0003] Current network slicing deployments mostly consider only the size of node resources, without comprehensively considering resource capacity, link length, and the operational status of physical network functions (PNFs). This makes it difficult to guarantee efficient utilization of different network needs and resources. Furthermore, physical network resources are constantly changing, and multiple network slice requirements can arise simultaneously. Current network slicing deployment and orchestration methods largely fail to consider the dynamic changes in physical network status and the adjustment of slice deployment strategies under multiple network slice requirements.

[0004] Therefore, how to overcome the shortcomings of existing technologies and solve the problem of difficulty in adjusting strategies during network slicing is a problem to be solved in this technical field. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention solves the problem of difficulty in adjusting the strategy during network slicing.

[0006] The embodiments of the present invention adopt the following technical solutions:

[0007] In a first aspect, this invention provides a method for digital twin network slicing based on reinforcement learning, specifically: a reinforcement learning module acquires current network information triples and network slice requirement triples from a knowledge graph. Each network slice corresponds to a set of network slice requirement triples. The current network information triples include all PNF connections in the network and the resource attributes of each PNF. The network slice requirement triples include all VNF connections for the slice requirement, the resource requirements of each VNF, and the slice priority. The reinforcement learning module generates at least one decision action for the network slice. Each decision action includes finding PNF nodes based on a node selection probability model and the knowledge graph, and mapping the resources allocated to each PNF node to a VNF as a resource allocation vector based on a resource allocation model. The knowledge graph makes decisions based on the decision actions to obtain decision results, which are the deployment positions of the network slice VNFs and network slice PNFs. The knowledge graph updates based on the decision results and transmits the updated network state back to the reinforcement learning module. After confirming that the network state is normal, the reinforcement learning module issues a network slice deployment strategy.

[0008] Preferably, the reinforcement learning module generates at least one decision action for a network slice, specifically including: based on the slice requirements and the knowledge graph, finding the PNF with the highest probability in the node selection probability model as the first node; obtaining a resource evaluation vector based on the knowledge graph, inputting it into the resource allocation model to allocate resources to the first VNF ​​node, and updating the knowledge graph; based on the position selected by the previous VNF node, performing knowledge reasoning based on the PNF connection relationships contained in the knowledge graph to find the possible PNF position selection for the next VNF ​​node, and so on until all nodes are deployed and allocated.

[0009] Preferably, finding the PNF with the highest probability in the node selection probability model as the first node specifically includes: finding all possible PNF positions in the action space of the first VNF ​​node and selecting them as the deployment action space of the first VNF ​​node, and obtaining the node state matrix; inputting the node state matrix into the node selection probability model, calculating the probability of each node being selected, and selecting the PNF with the highest probability as the first node; obtaining the resource evaluation vector of the first node based on the knowledge graph, and inputting the evaluation vector into the resource allocation model to allocate resources to the first VNF ​​node.

[0010] Preferably, the step of finding the PNF position selection of the possible next VNF ​​node specifically includes: based on the selected position of the previous VNF node, performing knowledge reasoning based on the PNF connection relationship contained in the knowledge graph to find all possible PNF position selections of the next VNF ​​node, which serve as the deployment action space for the next VNF ​​node; obtaining the node state matrix based on the knowledge graph, and sequentially deploying and allocating resources for the next node until all nodes have been deployed and allocated, thereby completing the selection of an entire slice deployment strategy based on the allocation results of all nodes.

[0011] Preferably, the knowledge graph is updated based on the decision result, specifically including: updating the knowledge graph according to the resource allocation strategy of reinforcement learning, and calculating the feedback value according to the feedback function; and updating the parameters of the node selection probability model using the gradient descent method in combination with the feedback value.

[0012] Preferably, the node selection probability model specifically includes: mapping the distance between PNFs, the resource requirements of slices for VNF nodes, and the resource status on PNFs into probabilities of possible selection, so as to select the PNF node with the highest probability of being acquired by the reinforcement learning module for deployment.

[0013] Preferably, the resource allocation model specifically includes: mapping the resource evaluation vector into a resource allocation vector through a function, so as to represent the computing, storage, and network resources allocated by the PNF node to the VNF.

[0014] Preferably, the reinforcement learning module confirming that the network status is normal further includes: when a network slice request is being deployed on a certain node, and a PNF with sufficient resources cannot be found due to excessive resource consumption by previous network slices, the resource allocation of each VNF in the first few network slices is proportionally reduced according to the priority of the network slices, provided that the resource allocation is not lower than the minimum requirement. The reduction in resource allocation is inversely proportional to the priority of the network slice.

[0015] Preferably, the reinforcement learning module confirming that the network state is normal further includes: when the network state changes and the current network state in the knowledge graph module cannot meet the slicing requirements of the original time, the reinforcement learning module reverts to the unmet slicing step and redeploys.

[0016] On the other hand, the present invention provides an apparatus for slicing a digital twin network based on reinforcement learning, specifically comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the method for slicing a digital twin network based on reinforcement learning in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows: using a reinforcement learning module to generate a network slice deployment strategy, quickly completing the decision on the time slice deployment scheme in a dynamic network environment, obtaining the optimal slice deployment scheme in the current network scenario, and better ensuring the efficient utilization of network resources. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 A schematic diagram of the network structure model for performing network slicing;

[0020] Figure 2 A flowchart of a method for slicing a digital twin network based on reinforcement learning, provided in an embodiment of the present invention;

[0021] Figure 3 A flowchart illustrating another method for slicing a digital twin network based on reinforcement learning, provided in an embodiment of the present invention;

[0022] Figure 4 A flowchart illustrating another method for slicing a digital twin network based on reinforcement learning, provided in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of the reinforcement learning module architecture provided in an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of a device structure for slicing a digital twin network based on reinforcement learning, provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0026] This invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly describe the functional logic relationship of each structural module, and do not limit the specific software and hardware implementation methods.

[0027] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] Example 1:

[0029] Reinforcement learning (RL) is an artificial intelligence technique that relies on continuous interaction with the environment to obtain feedback and optimize decisions accordingly. This patent combines knowledge graphs and reinforcement learning to implement network slice deployment decisions, thereby ensuring more efficient utilization of network resources.

[0030] To address the above issues, the method described in this patent implements network slicing through a reinforcement learning-based network slice deployment strategy generation method. The deployment strategy of the generated network slices can better guarantee the operational status of each network slice and the entire network.

[0031] like Figure 2 As shown in the figure, the specific steps of the digital twin network slicing method based on reinforcement learning provided in this embodiment of the invention are as follows:

[0032] Step 101: The reinforcement learning module obtains the current network information triples and network slice requirement triples in the knowledge graph. Each network slice corresponds to a set of network slice requirement triples.

[0033] The input to the reinforcement learning module comes from a knowledge graph, which consists of entity-relation-entity triples and entities and their associated attribute-value pairs. A triple can be represented as (Thing1, Relation, Thing2), where Thing1 represents the first entity, Thing2 represents the second entity, and Relation represents the relationship between Thing1 and Thing2. The method in this embodiment is used to calculate the deployment strategy of network slices based on the current network state; therefore, it requires the use of network information triples and network slice requirement triples. The current network information triple includes all PNF connection relationships in the network and the resource attributes of each PNF. Resource attributes include the determined hardware resource attributes or usable resource attributes of each PNF, such as computation, storage, and network resources. In the network information triple, Thing1 represents the PNF, Relation represents the resource type of the triple, and Thing2 represents the value of that type of resource required by Thing1. For example, in a specific scenario, a PNF (represented as PNF1) requires 100, 200, and 300 compute, storage, and network resources respectively. PNF1 and PNF2 have a connection relationship. This PNF can be represented as: (PNF1, Connection, PNF2), (PNF1, Compute, 100), (PNF1, Storage, 200), (PNF1, Network, 300). A network slice requirement triple includes all VNF connection relationships required for the slice, the resource requirements of each VNF, and the slice priority. Each network slice corresponds to a set of network slice requirement triples. In the network slice requirement triple, Thing1 represents the network slice or VNF, Relation represents the resource type of the triple, and Thing2 represents the value of that type of resource required by Thing1. Furthermore, when Thing1 is a network slice, the Relation type is "required VNF" or priority, and Thing2 is the VNF or priority value required by Thing1. For example, if network slice 1 requires VNF1, the triple is (slice 1, required, VNF 1). When Thing1 is a VNF, similar to the network information triple, the Relation is the type of computation, storage, network, etc., resources allocated to the VNF, and Thing2 is the value of that resource type. For example, if the computation resource allocation value of VNF1 is 100, the computation resource triple for VNF1 is (VNF1, computation, 100). In practical use, the reinforcement learning module may receive requests for multiple network slices simultaneously.

[0034] Step 102: The reinforcement learning module generates at least one decision action for a network slice. Each decision action includes finding PNF nodes based on the node selection probability model and the knowledge graph, and mapping the resources allocated to each PNF node to the VNF as a resource allocation vector based on the resource allocation model. The knowledge graph makes decisions based on the decision actions to obtain the decision results, which are the deployment locations of the network slice VNF and the network slice PNF.

[0035] The output of the reinforcement learning module is the deployment strategy for network slices, specifically which PNF each VNF in a network slice will be deployed on, and how many resources will be allocated to each VNF. The output of the reinforcement learning module is then fed into the knowledge graph, where it is updated.

[0036] Step 103: The knowledge graph is updated based on the decision results, and the updated network state is sent back to the reinforcement learning module. After the reinforcement learning module confirms that the network state is normal, it issues the deployment strategy for network slices.

[0037] In step 102, the VNFs required for a network slice change due to the decisions made by the reinforcement learning module. The computational, storage, and network resources required for each VNF also change. Therefore, the triples in the knowledge graph need to be updated accordingly based on the data after the decisions. For example, network slice 1 originally only needed VNF1, represented as (slice 1, required, VNF1). After the decision, it changes to require VNF1 and VNF2, and the triples are updated to (slice 1, required, VNF1) and (slice 1, required, VNF2). As another example, if the computational resource allocation value for VNF1 changes from 100 to 150, the corresponding triples also need to be updated from (VNF1, computation, 100) to (VNF1, computation, 150).

[0038] In the method provided in this embodiment, a network slice requirement is transformed into multiple VNF allocation requirements. The allocation process for each VNF first determines which PNF it will be allocated to, and then allocates the corresponding resources according to the requirement. After each VNF allocation is completed, a triple change occurs, and the next VNF ​​needs to be checked based on the new triple. Therefore, after each VNF allocation is completed, there is an interaction with the environmental knowledge graph.

[0039] The knowledge graph sends the updated network state back to the reinforcement learning module. The reinforcement learning module evaluates the adjusted network state and, after confirming that the network state is normal after using the slicing strategy, sends the deployment decision information to the control module for distribution.

[0040] After steps 101-103 provided in this embodiment, the reinforcement learning module can be used to optimize the network slice deployment decision based on the current network state, and the decision result can be confirmed to ensure that the network state is normal after the new slicing strategy is implemented.

[0041] Network slice deployment decisions based on reinforcement learning mainly consist of the following parts: environment, action, state, reward, and decision model. The environment for reinforcement learning is the network knowledge graph. The state refers to the current state of the network. An action is a decision made by reinforcement learning in the current state, namely, a node selection scheme and a resource allocation scheme for a network slice. The reward is the feedback given by the environment after reinforcement learning makes an action, reflecting the quality of the slice deployment decision. The decision model in this patent includes a node selection probability model and a resource allocation model. Reinforcement learning generates a slice deployment decision action based on the decision model.

[0042] like Figure 3 As shown, the following steps can be used to complete the deployment strategy for generating network slices using the reinforcement learning module.

[0043] Step 201: Based on the slicing requirements and the knowledge graph, find the PNF with the highest probability in the node selection probability model and select it as the first node.

[0044] Based on the slicing requirements and using the knowledge graph, all possible PNF locations in the action space of the first VNF ​​node are identified and selected as the deployment action space for the first VNF ​​node, resulting in a node state matrix. This node state matrix is ​​then input into a node selection probability model to calculate the probability of each node being selected, and the PNF with the highest probability is chosen as the first node. Finally, based on the knowledge graph, a resource evaluation vector for the first node is obtained, and this vector is input into a resource allocation model to allocate resources to the first VNF ​​node.

[0045] Step 202: Obtain the resource evaluation vector based on the knowledge graph, input it into the resource allocation model to allocate resources to the first VNF ​​node, and update the knowledge graph.

[0046] After a node is deployed, the knowledge graph is updated according to the resource allocation strategy of reinforcement learning, and the feedback value is calculated according to the feedback function.

[0047] During the update process, the parameters of the node selection probability model also need to be updated using gradient descent, taking into account the feedback values. The specific calculation formula is as follows.

[0048]

[0049] This formula represents the current node selection probability model after gradient descent based on the feedback value for each node selection probability in the node selection probability model.

[0050] Step 203: Based on the position selected by the previous VNF node, perform knowledge reasoning based on the PNF connection relationship contained in the knowledge graph to find the possible PNF position selection of the next VNF ​​node, and search in sequence until all nodes are deployed and allocated.

[0051] After selecting the first VNF ​​node, it is necessary to sequentially obtain the PNF positions of all subsequent VNF ​​nodes. Based on the selected position of the previous VNF node, knowledge reasoning is performed using the PNF connections contained in the knowledge graph to find the PNF position selection of all possible next VNF ​​nodes, which serves as the deployment action space for the next VNF ​​node. Then, based on the knowledge graph, the node state matrix is ​​obtained, and the previous steps are repeated to sequentially deploy and allocate resources for the next node until all nodes are deployed and allocated. The allocation results of all nodes complete the selection of the entire slice deployment strategy.

[0052] In the actual decision-making process, if no node position option is found when selecting the deployment strategy for the k-th node, then we return to the (k-1)-th node and select the option with the second highest probability before proceeding to the next step.

[0053] When multiple network slices are required, a deployment policy is generated for one network slice request before making a deployment decision for the next network slice.

[0054] After steps 201-203 provided in this embodiment, the network slicing decision in step 102 can be completed, and the deployment strategies for all VNFs and PNFs can be obtained. Combined with the aforementioned steps, the complete process of network slicing and adjustment is as follows: Figure 4 As shown.

[0055] Furthermore, after making all network slice deployment decisions according to the above steps, the deployment decisions need to be transmitted to the knowledge graph module for updating. The knowledge graph then transmits the updated network state back to the reinforcement learning module. This iterative feedback process improves decision accuracy and ensures that the correspondences in the knowledge graph can adapt to dynamic changes in the network environment.

[0056] In practical implementation scenarios, reinforcement learning models can be built, deployed, and feedback completed in the following ways. Specifically, methods such as... Figure 5 The reinforcement learning module shown.

[0057] The network operational status of a network slice specifically includes the location, resources, and relationship information of the network slice PNF and network slice VNF. In this embodiment, the following mathematical model is used to describe the network operational status: This represents the PNF in the physical cluster layer. This represents the physical link between PNFs in the physical cluster layer, for example, Indicates slice and The physical links between them. Network slices existing in the network are denoted by S = {S1, S2, ..., S...}. b} indicates that there are a total of b network slices. In a network slice S s middle, Indicates a VNF node. Represents the virtual links between VNF nodes, for example Indicates slice and Virtual links between them. For The computing, storage, and network resources are respectively represented as VNF nodes in slices The computing, storage, and network resources are respectively represented as

[0058] The reliability of each PNF or VNF is measured by Mean Time To Failure (MTTF), and its serviceability is measured by Mean Time To Restoration (MTTR). Therefore, the availability A of a PNF (VNF) is... PNF (A VNF It can be expressed using the following formula.

[0059]

[0060] The availability of the entire slice is equal to the minimum availability of the PNF and VNF that make up the slice, and is used to evaluate the network's operational status.

[0061] The positional relationship between VNF and PNF is used It can be expressed by the following formula:

[0062]

[0063] The above formula means: if exist Above, then Otherwise, it is 0.

[0064] and The computing, storage, and network resources should meet the following conditions:

[0065]

[0066] The above formula means that the network operating state is defined based on the availability of PNF and VNF and resource conditions, as follows: When slicing the network, select the VNF and the deployed PNF, and sort the PNFs according to their spatial location.

[0067] In a specific implementation scenario of this embodiment, the total number of PNFs is k, and the set of all PNFs is represented as PNF(PNF1, PNF2, ..., PNF...). k The total number of VNFs is r, and the set of all VNFs is represented as VNF(VNF1, VNF2, ..., VNF). r ).

[0068] The initial state of a PNF / VNF consists of four parts: The first part represents the sequence number of the previous PNF / VNF, with the first bit of the first PNF / VNF being 0. The second part represents the sequence number of the PNF to which the PNF / VNF resides, with the second part of the first PNF / VNF being 1. The number of bits in the first two parts varies depending on the number of PNFs / VNFs. For example, if there are 4 PNFs, 2 bits are used; if there are 10 PNFs, 4 bits are needed. The third part represents the computation, storage, and network resources in the PNF / VNF. For a PNF, 1 indicates that computation / storage / network resources are available, and 0 indicates that they are used up. For a VNF, 1 indicates that computation / storage / network resources are available in the current PNF, and 0 indicates that they are used up. The fourth part represents the operating status of the PNF / VNF, determined by the availability defined above. When the calculated availability is higher than the set availability threshold, it is considered normal operation (1), and abnormal operation (0). For example, when there are 2 PNFs and 3 VNFs, where VNF1 and VNF2 are deployed on PNF1 and VNF3 is deployed on PNF2, all are running normally with sufficient resources. For each state value, the first two parts have a maximum value of 3 (3 VNFs) each occupying 2 bits, the third part occupies 3 bits, and the fourth part occupies 1 bit. Therefore, the initial states of the PNFs are represented as 00011111 and 01101111, respectively. The initial states of the VNFs are represented as 00011111, 01011111, and 10101111. For example, in the PNF initial state 00011111, starting from the first bit: the first part 00 represents the sequence number of the previous PNF; the second part 01 represents the sequence number of the PNF; each bit in the third part 111 corresponds to the computation, storage, and network resources of the VNF, with a value of 1 indicating that the corresponding resources are available; the fourth part 1 indicates that the VNF is running normally.

[0069] In reinforcement learning modules, the action space for network slice deployment is the set of node deployment strategies that a node in a slice can choose. An action represents a node deployment strategy, which can be represented as A = (a1, a2). Here, a1 represents selecting a PNF node, and a2 represents the amount of resources allocated to that node. If there are k possible actions for deploying a node, then the action space can be represented as A s =(A1,A2,…,A) k The action space is constrained by network resource limitations.

[0070] The node selection probability model is used to select a node for deployment. It maps the distance between PNFs, the resource requirements of slices for VNF nodes, and the resource state on the PNFs to probabilities of possible selection, thus enabling the reinforcement learning module to choose the PNF node with the highest probability for deployment.

[0071] The input to the node selection probability model is a node state matrix X. Each row of the matrix is ​​the node state vector of each PNF node. in, The selectivity of node i in PNF is determined by whether it has been selected in the previous few hops and whether it is reachable. Selectable is 1, and unselectable is 0. This represents the physical distance of the shortest available link from PNF node i to the previous PNF node. If it is the first node, then... Both are 1. If there is no link from PNF node i to the previous PNF node, then Both are -1. These represent the degree to which the compute, storage, and network resources on PNF node i meet the demand, respectively. They are obtained by calculating the difference between the available resources on PNF node i and the minimum resource requirement of the slice for that node. Assuming a node's resource requirement is (10, 10, 10), PNF... i The available resources are (100, 100, 90), and the first few jumps were not selected. (PNF) i If the physical distance to the previous PNF node is 3, then PNF i The corresponding node state vector is represented as X. i =[1,3,90,90,80].

[0072] The node probabilistic selection model first performs a weighted calculation on the node state matrix: Y = WX + b, which yields the evaluation score Y = (y1, y2, ... y) for each PNF node to determine whether it can serve as the next PNF node. n Then, based on the evaluation score, the probability of each PNF node being selected is calculated. This patent uses a neural network to parameterize the node selection probability model, where W is the neuron weight parameter and b is the model bias parameter, both of which are obtained from prior model training.

[0073] The input to the resource allocation model is a resource evaluation vector. in This represents the minimum requirements for VNF computing, storage, and network resources. This indicates the availability of compute, storage, and network resources on a PNF node. This indicates the priority of network slices. The resource model maps resource evaluation vectors to resource allocation vectors through a function. The resource allocation vector can be represented as: in, These represent the compute, storage, and network resources allocated by the PNF node to the VNF, respectively. In practice, appropriate parameters can be selected as the parameters of the resource allocation vector according to actual needs. The resource allocation model is represented by y. r =f(x) r The mapping function f(·) can be fitted by a neural network.

[0074] The feedback function is used to evaluate the effectiveness of an action and deployment strategy, providing a reward. When selecting a policy action, the reinforcement learning model updates its policy model based on the reward received from the previous action. The goal of reinforcement learning is to select the slice deployment strategy with the highest reward. The feedback function consists of three parts: deployment accuracy, slice requirement satisfaction, and link efficiency. Deployment accuracy indicates whether the selected VNF, PNF, and link can function properly. For example, deployment accuracy is 0 when the resource usage of the selected VNF node exceeds the resource limit of its PNF; deployment accuracy is also 0 when the selected VNF or PNF is not functioning correctly.

[0075]

[0076] Slice demand satisfaction refers to the degree to which the resources allocated to a slice meet business requirements. Consider two scenarios: one where the computing, storage, and network resources allocated to a slice are insufficient to meet business needs; and another where the resources allocated to a slice can meet the requirements, but excessive allocation may lead to low network resource utilization. Therefore, slice demand satisfaction can be expressed by the following formula.

[0077]

[0078] Where α represents the feedback parameter when demand is not met, x represents the number of resources that a node cannot satisfy, γ represents the feedback parameter when resources are over-allocated, and u d This represents the resource utilization rate of a node's VNF, a. d This indicates the resources allocated to this node.

[0079] Link efficiency reflects the distance between the PNFs deployed on two adjacent VNFs. The PNF nodes of two adjacent VNFs should be as close as possible; excessive distance will increase latency. Link efficiency can be expressed by the following formula.

[0080]

[0081] The feedback function, after combining the three parts, can be defined as:

[0082] r = r ACCURACY (ηr SATISFACATION +ξr EFFICIENCY (8)

[0083] Where η and ξ are the proportional coefficients of slice demand satisfaction and link efficiency, respectively. r is the feedback value, calculated as the ratio of demand satisfaction to r. SATISFACATION Link efficiency r EFFICIENCY Weighted summation, then summed with deployment accuracy r ACCURACY Multiply. As can be seen from this formula, the higher the slice demand satisfaction and link efficiency, the higher the feedback value of the slice deployment strategy. The parameters in the feedback function can be fitted using a neural network.

[0084] Using the reinforcement learning module described above, along with the corresponding feedback function, the deployment of each VNF and PNF in the network slice can be completed based on the current network state obtained.

[0085] Furthermore, for multiple network slices in a digital twin reinforcement learning module, network resource conflicts may arise, requiring strategy adjustments. The following provides two common strategy adjustment methods; in specific implementation scenarios, you can refer to these methods for strategy adjustments based on actual needs.

[0086] (1) When a network slice request is being deployed on a node, and a PNF with sufficient resources cannot be found due to excessive resource consumption by previous network slices, the resources of each VNF in the first few network slices are proportionally reduced according to the priority of the network slices, provided that the resource requirements are not lower than the minimum requirements. The reduction in resource size is inversely proportional to the priority of the network slice. The specific reduction in resource size v can be calculated using the following formula.

[0087]

[0088] Where i represents the slice priority order, z represents the previously allocated resource size, h represents the minimum slice requirement, and the parameter θ can take values ​​in (0,1]. The larger θ is, the greater the degree of resource reduction. In the preferred scheme, when there is a large demand for slices in the network, θ can be taken from 0.5 to 1.

[0089] (2) When the network state changes and the current network state in the knowledge graph module cannot meet the slice requirements of the original state, in order to avoid this resource conflict, after generating a single network slice deployment strategy, the deployment strategy is passed to the knowledge graph module for testing to determine whether the current network state can meet the already generated network slice decision. If it does not meet the requirements, the reinforcement learning module reverts to the unmet slice step and re-deploys the decision; if it does meet the requirements, the network state is updated, and the next deployment decision is made based on the updated network state.

[0090] By using the above-mentioned strategy adjustment methods, adjustments can be made quickly and effectively in the event of decision-making errors, ensuring that the issued deployment strategies can maintain normal network status.

[0091] The reinforcement learning-based digital twin network slicing method provided in this embodiment incorporates network operating status, resource utilization, and link efficiency during node selection and resource allocation. This allows for the generation of reasonable deployment strategies for network slices while ensuring network operating status. Under multiple network slice requirements, the deployment strategy for network slices is dynamically adjusted based on their priority and network resource status, thereby ensuring the stable operation of each network slice and the entire network.

[0092] Example 2:

[0093] Based on the reinforcement learning-based digital twin network slicing method provided in Embodiment 1, in different specific application scenarios and in certain specific implementations, the reinforcement learning-based digital twin network slicing method provided in Embodiment 1 can be implemented through the specific implementations in this embodiment, so that the user's task requirements are transformed into network slicing requirements through the knowledge graph module.

[0094] In a specific implementation scenario, there are 6 PNFs in the network, with computing, storage, and network resources of PNF1 (100, 200, 95), PNF2 (100, 60, 70), PNF3 (200, 100, 80), PNF4 (100, 200, 95), PNF5 (200, 200, 200), and PNF6 (100, 100, 100). PNF1 is connected to PNF3 and PNF6, and PNF2 is connected to PNF4 and PNF5. They are physically reachable.

[0095] The user initiates two slice requests: Network slice 1 requires three slices. The minimum computation, storage, and network requirements for each VNF are as follows: The slice priority is 1. Network slice 2 requires 4 VNFs, and the computation, storage, and network requirements of each VNF are as follows: The slice priority is 2. The superscript of a VNF indicates the network slice it belongs to, and the subscript is the VNF's own index within each network slice.

[0096] The knowledge graph module constructs the current network knowledge graph based on the previous network state and available resources, and passes the user's slice requirements and network knowledge graph information to the reinforcement learning module in the form of triples. The network information triples are: (PNF1, connection, PNF3), (PNF1, computation, 100), (PNF1, storage, 100), (PNF1, network, 90), (PNF2, computation, 100), (PNF2, storage, 60), (PNF2, network, 70), (PNF3, computation, 200), (PNF3, storage, 100), (PNF3, network, 80), (PNF4, computation, 100), (PNF4, storage, 200), (PNF4, network, 95). The remaining triples can be constructed according to the same rules.

[0097] The slice requirement triple for network slice 1 is: (slice 1, priority is 1),

[0098] (Slice 1, required) (Slice 1, required) (Slice 1, required) ), ( Calculate, 10), ( Storage, 10), ), ( Network, 10), (

[0099] Calculate, 10), ( Storage, 20), ( Network, 10), ( Calculate, 20), ( Storage, 10), ( Network, 20).

[0100] A set of slice requirement triples for network slice 2 is (slice 2, priority is, 2), (slice 2, requires, (Slice 2, required) (Slice 2, required) ),

[0101] (Slice 2, required) ), ( Calculate, 20), ( Storage, 20), ),

[0102] ( Network, 20), ( Calculate, 20), ( Storage, 20), ( Network, 10),

[0103] ( Calculate, 10), ( Storage, 10), ( Network, 20), ( Calculate, 30),

[0104] ( Storage, 10), ( Network, 10).

[0105] Following step 101, the knowledge graph transmits the aforementioned network information triples and network slice request triples, along with the network information and network slice requests, to the reinforcement learning module. The reinforcement learning module then sequentially makes slice deployment decisions for network slice 1 and network slice 2, respectively.

[0106] Following step 102, based on the knowledge graph, first... The node deployment constructs a node state matrix X. Each row of the matrix is ​​the node state vector of each PNF node. in, The selectivity of node i in PNF is determined by whether it has been selected in the previous few hops and whether it is reachable. Selectable is 1, and unselectable is 0. This represents the physical distance of the shortest available link from PNF node i to the previous PNF node. If it is the first node, then... All are 1. These represent the degree to which computing, storage, and network resources on PNF node i meet the demand, respectively. This is obtained by calculating the difference between the available resources on PNF node i and the minimum resource requirement of the slice for that node. Assume we have PNF1: (100, 100, 90) and NF2: (100, 60, 70). For the first VNF, both are selectable, so the corresponding node state vectors are X1 = [1, 1, 90, 90, 80] and X2 = [1, 1, 90, 60, 50]. After constructing the node state matrix, it is input into the node selection probability model. The node probability selection model is a neural network model. The model first performs a weighted calculation on the node state matrix to obtain the evaluation score of each PNF node, and then calculates the probability of each PNF node being selected based on its evaluation score. The calculated selection probability for each PNF is [0.9, 0.7, ...], where PNF1 has the highest probability of 0.9, so it is selected. Deployed on PNF1. Then, construct the resource evaluation vector as x. rThe input vector (10, 10, 10, 100, 100, 90) is used to calculate the resources to be allocated in the resource allocation model. The model maps the resource evaluation vector to a resource allocation vector using a function, which is pre-fitted using a neural network. If the calculated output is (85, 85, 80), then... The allocated computing, storage, and network resources are 85, 85, and 80 respectively. The amount of resources allocated depends on the resource allocation model. If the model is trained well, the allocation result will be more reasonable, and no further adjustments are needed. If the model is not trained well enough, the most reasonable partitioning cannot be obtained, and further adjustments will be required.

[0107] After a node deployment is completed, the environmental knowledge graph is updated, such as reducing the available resources of PNF1 to (15, 15, 10). Next, the deployment accuracy, slice demand satisfaction, and link efficiency of the current slice are calculated according to the calculation formula mentioned above. Then, the reward for this network slice resource allocation action is calculated by combining the feedback function formula. Since the node was successfully deployed, the deployment accuracy is calculated as 1. For slice demand satisfaction, assuming the feedback coefficient is 2, the value can be calculated as 2x(10 / 85+10 / 85+10 / 80) / 3=0.24 according to the formula (5) based on the node demand and the allocated resources. Since this node is the first node, the link efficiency is calculated as 1. Assuming the ratio coefficients of slice demand satisfaction and link efficiency are 0.5 respectively, the feedback reward value can be calculated as 1x(0.5*0.24+0.5*1)=0.62 according to the formula (7). Then, the parameters of the node selection probability model are updated by using the gradient descent method in combination with the feedback value. Since this network slice resource allocation action successfully allocated resources to a network slice and received a reward, it indicates that the resource allocation model's mapping direction was correct. Therefore, the result of the parameter update is to encourage the model to continue optimizing in this mapping direction.

[0108] Next, based on the selected position of the first VNF ​​node, PNF1, we will... The node deployment constructs a node state matrix X. Assume PNF3:(200,100,80), NF4:(100,200,95), the physical distance from PNF3 to PNF1 is 3, and PNF4 is unreachable from PNF1. The corresponding node state vectors are X3 = [1,3,190,80,70] and X4 = [0,-1,90,180,85]. After constructing the node state matrix, it is input into the node selection probability model to calculate the selection probability of each PNF as [...0.8,0.6,…]. PNF3 has the highest probability of 0.8, so it is selected. Deployed on PNF3. Then, construct the resource evaluation vector as x. rThe input (10, 20, 10, 200, 100, 80) to the resource allocation model yields an output (150, 80, 50). Allocate computing, storage, and network resources of 150, 80, and 50 respectively.

[0109] After deploying the second node, the environmental knowledge graph is updated, such as reducing PNF3's available resources to (50, 20, 30). Then, the parameters of the reward update node selection probability model are calculated based on the feedback function. The calculation method is as follows:

[0110]

[0111] Where w is the model parameter, η is the update coefficient, and r is the feedback function value.

[0112] Repeat the above steps for the third node. Deployment, node selection probability model selection Deployed on PNF6, the resource allocation model allocates 60, 50, and 60 compute, storage, and network resources respectively, thus completing the deployment decision for the first network slice request. After completing the deployment decision for network slice 1, the reinforcement learning module begins the deployment decision for the second network slice. When it becomes the first node of the second network slice... When making deployment decisions, it was found that only PNF1 was available, but PNF1 was not selected in the first network slice. Allocating too many resources leads to insufficient resources. At this point, the resource allocation adjustment formula (9) is used to calculate the required reduction in resources (42, 42, 40) based on the priority of the first network slice. Then, adjustments are made... The resource allocation was reduced from (85,85,80) to (43,43,40). Now the available resources on PNF1 have changed from (15,15,10) to (57,57,50), which can satisfy... The resource requirements were met. Therefore, the deployment decision for the second network slice was made following the previous steps.

[0113] After making deployment decisions for each node in the second network slice, the deployment decisions for these two slices are as follows: and The deployment decision is transmitted to the knowledge graph module, which then updates the knowledge graph module.

[0114] Following step 103, the knowledge graph transmits the updated network information back to the reinforcement learning module. After obtaining the average failure time and average repair time of each PNF from the network information, the availability of each PNF is calculated according to formula (1). Assuming that the calculated availability of PNF1 is 0.99, PNF2 is 0.98, PNF3 is 0.96, PNF4 is 0.97, and PNF6 is 0.98, and the set availability threshold is 0.95, then all PNFs are available. Based on the availability of PNFs and the resource conditions of PNFs and VNFs, the network operating status of the slice can be obtained. Specifically, the PNF states of slice S1 are 0000001111, 0000101111, and 0101001111, respectively, and the initial VNF states of slice S1 are 000001111, 010101111, and 101001111, respectively; the PNF states of slice S2 are 0000011111, 0010111111, and 0111011111, respectively, and the initial VNF states of slice S2 are 000011111, 010111111, 101011111, and 111011111, respectively. Therefore, the network status is assessed as normal, and the deployment decision information is then transmitted to the control module for distribution, completing the user's network slice request.

[0115] As can be seen from the above specific examples, the method for digital twin network slicing based on reinforcement learning provided in Example 1 can quickly and effectively complete the digital twin network slicing, better ensure the efficient utilization of network resources, and ensure that the network status remains normal under dynamic network changes.

[0116] Example 3:

[0117] Based on the reinforcement learning-based digital twin network slicing method provided in Embodiments 1 and 2 above, this invention also provides an apparatus for implementing the above method using reinforcement learning-based digital twin network slicing, as shown in the figure, which is a schematic diagram of the apparatus architecture of an embodiment of this invention. The apparatus for reinforcement learning-based digital twin network slicing in this embodiment includes one or more processors 11 and a memory 12. Figure 6 Take a processor 11 as an example.

[0118] Processor 11 and memory 12 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0119] The memory 12 serves as a non-volatile computer-readable storage medium for the reinforcement learning-based digital twin network slicing method. It can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the reinforcement learning-based digital twin network slicing methods in Embodiments 1 and 2. The processor 11 executes various functional applications and data processing of the reinforcement learning-based digital twin network slicing apparatus by running the non-volatile software programs, instructions, and modules stored in the memory 12, thereby implementing the reinforcement learning-based digital twin network slicing methods of Embodiments 1 and 2.

[0120] Memory 12 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 12 may optionally include memory remotely located relative to processor 11, which can be connected to processor 11 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] The program instructions / modules are stored in memory 12. When executed by one or more processors 11, they perform the reinforcement learning-based digital twin network slicing methods described in Embodiments 1 and 2 above, for example, performing the methods described above. Figure 2 and Figure 3 The steps shown.

[0122] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for slicing digital twin networks based on reinforcement learning, characterized in that, Specifically, it includes: The reinforcement learning module obtains the current network information triplet and network slice requirement triplet from the knowledge graph. Each network slice corresponds to a set of network slice requirement triplets. The current network information triplet includes all PNF connection relationships in the network and the resource attributes of each PNF. The network slice requirement triplet includes all VNF connection relationships of the slice requirement, the resource requirements of each VNF, and the slice priority. The reinforcement learning module generates at least one decision action for a network slice. Each decision action includes finding PNF nodes based on the node selection probability model and the knowledge graph, and mapping the resources allocated to each PNF node to the VNF as a resource allocation vector based on the resource allocation model. The knowledge graph makes decisions based on the decision actions to obtain the decision results, which are the deployment positions of the network slice VNF and the network slice PNF. The knowledge graph is updated based on the decision results, and the updated network state is sent back to the reinforcement learning module. After the reinforcement learning module confirms that the network state is normal, it issues the deployment strategy for network slices. This includes: updating the knowledge graph according to the resource allocation strategy of reinforcement learning and calculating the feedback value according to the feedback function; and updating the parameters of the node selection probability model using gradient descent based on the feedback value. The method further includes: when a network slice request is being deployed on a certain node, and a PNF that cannot meet the resource requirements cannot be found due to the excessive resource consumption of previous network slices, the resources of each VNF in the first few network slices are proportionally reduced according to the priority of the network slices, provided that the requirements are not lower than the minimum requirements. The reduction in resource size is inversely proportional to the priority of the network slice, so as to make quick and effective adjustments in case of decision errors and ensure that the issued deployment strategy can maintain the normal network status. Reduced resource size Calculate using the following formula: in, For slice priority order, The size of the resources previously allocated. Minimum requirements for slicing, parameters It can take values ​​in (0,1].

2. The method for slicing digital twin networks based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning module generates decision actions for at least one network slice, specifically including: Based on the slicing requirements and the knowledge graph, the PNF with the highest probability in the probability model is selected as the first node. The resource evaluation vector is obtained based on the knowledge graph and input into the resource allocation model to allocate resources to the first VNF ​​node, and the knowledge graph is updated. Based on the position selected by the previous VNF node, knowledge reasoning is performed based on the PNF connection relationships contained in the knowledge graph to find the possible PNF position selection of the next VNF ​​node, and so on until all nodes are deployed and allocated.

3. The method for slicing digital twin networks based on reinforcement learning according to claim 2, characterized in that, The process of finding the node by selecting the PNF with the highest probability in the probability model as the first node specifically includes: Find all possible PNF positions in the action space of the first VNF ​​node and select them as the deployment action space of the first VNF ​​node, and obtain the node state matrix. Input the node state matrix into the node selection probability model, calculate the probability of each node being selected, and select the PNF with the highest probability as the first node; The resource evaluation vector of the first node is obtained based on the knowledge graph. The evaluation vector is then input into the resource allocation model to allocate resources to the first VNF ​​node.

4. The method for slicing digital twin networks based on reinforcement learning according to claim 2, characterized in that, The selection of the PNF position for finding the possible next VNF ​​node specifically includes: Based on the selected position of the previous VNF node, knowledge reasoning is performed based on the PNF connection relationships contained in the knowledge graph to find all possible PNF position selections for the next VNF ​​node, which serve as the deployment action space for the next VNF ​​node. Based on the knowledge graph, a node state matrix is ​​obtained. The deployment and resource allocation of the next node are carried out sequentially until all nodes are deployed and allocated. The allocation results of all nodes complete the selection of the entire slice deployment strategy.

5. The method for slicing digital twin networks based on reinforcement learning according to claim 1, characterized in that, The node selection probability model specifically includes: The distance between PNFs, the resource requirements of slices for VNF nodes, and the resource status on PNFs are mapped to the probabilities of possible selections, so as to select the PNF node with the highest probability of reinforcement learning modules for deployment.

6. The method for slicing digital twin networks based on reinforcement learning according to claim 1, characterized in that, The resource allocation model specifically includes: The resource evaluation vector is mapped to the resource allocation vector through a function, so as to represent the computing, storage and network resources allocated by the PNF node to the VNF.

7. The method for slicing digital twin networks based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning module confirms that the network status is normal and also includes: When the network state changes and the current network state in the knowledge graph module can no longer meet the slicing requirements of the original time step, the reinforcement learning module reverts to the unmet slicing step and redeploys the decision.

8. A device for slicing digital twin networks based on reinforcement learning, characterized in that: The method includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. When executed by the processor, the instructions are used to perform the method for slicing a digital twin network based on reinforcement learning as described in any one of claims 1-7.

Images