End-to-end deterministic slice arrangement method and device based on deep reinforcement learning

Abstract

The invention discloses an end-to-end deterministic slice arrangement method and device based on deep reinforcement learning, and the method comprises the steps: mapping a physical network node to a virtual network through a virtualization technology, and obtaining an end-to-end network slice virtualization model; obtaining resource data of the target network slice, and calculating an information freshness index, a reliability index, a time delay index and a throughput index of the target network slice according to the resource data; constructing a resource allocation optimization model according to the information freshness index, the reliability index, the time delay index and the throughput index of the target network slice; performing optimization solution on the resource allocation optimization model through a deep reinforcement learning algorithm and a Markov decision process model to obtain a resource allocation scheme of the target network slice; and allocating resources to different services according to the resource allocation scheme. Therefore, high service satisfaction is kept under different network conditions, deterministic transmission of power services is realized, communication time delay is reduced, and reliability is improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular to an end-to-end deterministic slice orchestration method and device based on deep reinforcement learning. Background Art

[0002] In the power industry, the introduction of 5G private networks can effectively support the communication needs of various business scenarios in smart grids, such as remote power equipment monitoring, smart meter data collection, load scheduling, fault detection and early warning, etc. In order to ensure the power industry's high reliability, low latency and deterministic service requirements for data transmission, 5G power virtual private networks are widely used, providing end-to-end customized communication services for different businesses through network slicing technology.

[0003] As one of the core technologies of 5G networks, network slicing technology virtualizes physical network resources into multiple independent logical network slices to meet the quality of service (QoS) requirements of different services. However, in the power industry, due to the criticality of power services and high requirements for communication quality, especially ultra-reliable and low-latency communication (URLLC) services, more precise network resource management and slice orchestration algorithms are required to ensure the certainty of end-to-end communication. Traditional static resource allocation methods are difficult to cope with dynamic network environments and complex business needs, resulting in waste of resources or failure to meet business needs in a timely manner. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an end-to-end deterministic slice orchestration method and device based on deep reinforcement learning to optimize network resource allocation and meet the deterministic requirements of power business for information freshness, reliability, latency and throughput.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] An end-to-end deterministic slice orchestration method based on deep reinforcement learning, comprising:

[0007] Map physical network nodes to virtual networks through virtualization technology to obtain an end-to-end network slicing virtualization model;

[0008] Obtain resource data of the target network slice, and calculate the information freshness index, reliability index, latency index, and throughput index of the target network slice according to the resource data;

[0009] Building a resource allocation optimization model according to the information freshness index, reliability index, latency index and throughput index of the target network slice;

[0010] The resource allocation optimization model is optimized and solved by a deep reinforcement learning algorithm and a Markov decision process model to obtain a resource allocation solution for the target network slice;

[0011] Resources are allocated to different services according to the resource allocation scheme.

[0012] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0013] An end-to-end deterministic slice orchestration device based on deep reinforcement learning includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned end-to-end deterministic slice orchestration method based on deep reinforcement learning is implemented.

[0014] The beneficial effects of the present invention are as follows: by constructing a resource allocation optimization model including information freshness index, reliability index, latency index and throughput index, it is ensured that the deterministic needs of the power business are met; at the same time, compared with the traditional fixed resource allocation strategy, the introduction of the deep reinforcement learning algorithm can continuously adjust the resource allocation of network slices, and continuously maintain a high level of business satisfaction under different network conditions, thereby realizing deterministic transmission of the power business, reducing communication latency and improving reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flowchart of the steps of an end-to-end deterministic slice orchestration method based on deep reinforcement learning in an embodiment of the present invention;

[0016] Figure 2 A schematic diagram of an end-to-end network slicing architecture of a 5G power virtual private network of an end-to-end deterministic slicing orchestration method based on deep reinforcement learning in an embodiment of the present invention;

[0017] Figure 3 A schematic diagram of a deterministic service resource allocation method based on deep reinforcement learning of an end-to-end deterministic slice orchestration method based on deep reinforcement learning in an embodiment of the present invention;

[0018] Figure 4 Schematic diagram of the structure of an end-to-end deterministic slice orchestration device based on deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in conjunction with the implementation modes and the accompanying drawings.

[0020] An end-to-end deterministic slice orchestration method based on deep reinforcement learning, comprising:

[0021] Map physical network nodes to virtual networks through virtualization technology to obtain an end-to-end network slicing virtualization model;

[0022] Obtain resource data of the target network slice, and calculate the information freshness index, reliability index, latency index, and throughput index of the target network slice according to the resource data;

[0023] Building a resource allocation optimization model according to the information freshness index, reliability index, latency index and throughput index of the target network slice;

[0024] The resource allocation optimization model is optimized and solved by a deep reinforcement learning algorithm and a Markov decision process model to obtain a resource allocation solution for the target network slice;

[0025] Resources are allocated to different services according to the resource allocation scheme.

[0026] From the above description, it can be seen that the beneficial effects of the present invention are: by constructing a resource allocation optimization model including information freshness index, reliability index, latency index and throughput index, it is ensured that the deterministic needs of the power business are met; at the same time, compared with the traditional fixed resource allocation strategy, the introduction of deep reinforcement learning algorithm can continuously adjust the resource allocation of network slices, and can continuously maintain a high level of business satisfaction under different network conditions, thereby realizing deterministic transmission of power business, reducing communication latency and improving reliability.

[0027] Further, the calculating the information freshness index, reliability index, latency index and throughput index of the target network slice according to the resource data includes:

[0028] Obtaining a parameter value of a target parameter to be calculated and a preset comparison value corresponding to the target parameter;

[0029] The target parameters include the information freshness, reliability, latency and throughput;

[0030] Obtaining an evaluation value of the target parameter according to a ratio of the parameter value to the preset comparison value;

[0031] Obtain the evaluation values ​​of all the target parameters under the target network slice and calculate the average value to obtain the target indicator corresponding to the target parameter.

[0032] From the above description, it can be seen that after obtaining the parameter values ​​of information freshness, reliability, latency and throughput, the parameter values ​​are compared with the preset comparison values ​​to determine whether the parameter values ​​such as the information freshness of the service meet the transmission requirements, and the average value of the evaluation values ​​of all target parameters under the target network slice is used as the corresponding indicator to achieve the evaluation of each indicator.

[0033] Further, calculating the information freshness index of the target network slice includes:

[0034]

[0035] Among them, a m (t+1) represents the information freshness of business m at time t+1; binary variable Indicates that the power virtual private network device chooses to successfully transmit data in time slot t. If so, then otherwise, d(t) represents the end-to-end delay of data transmission in time slot t of the power virtual private network equipment; a0 is a fixed constant preset by the system; a(t) represents the average information freshness; f A (t) represents the information freshness index evaluation function, A m represents the preset timeliness index of the transmission service m; M t represents the total number of services in time slot t, E A (t) represents the information freshness index.

[0036] From the above description, it can be seen that the above function can describe the information freshness of the business at different times, and by setting a fixed constant a0, the information freshness will not stay at the same value for a long time when no new data arrives, thereby better reflecting the timeliness of the information.

[0037] Further, calculating the reliability index of the target network slice includes:

[0038]

[0039] p soft,u ∈(0,1);

[0040] p phy,i ∈(0,1);

[0041]

[0042]

[0043] Among them, R m (t) represents the reliability of service m at time t; u is a virtual node; A set of nodes representing business m; binary variable Indicates whether the underlying service node i used to transmit service m is mapped to virtual node u. If so, then otherwise p soft,u ∈(0,1) represents the probability that software node u works normally; p phy,i ∈(0,1) represents the probability that physical node i works normally; f R (t) represents the reliability index evaluation function, R m Represents the preset reliability index of transmission service m; M t represents the total number of services in time slot t, E R (t) represents the reliability index.

[0044] From the above description, it can be seen that by considering the probability of the virtual node working normally and the probability of the physical node working normally, and the reliability model adjusts the probability according to the time change, it can more accurately reflect the dynamic reliability of the node or link.

[0045] Further, calculating the latency index of the target network slice includes:

[0046]

[0047] Where d(t) represents the end-to-end delay of data transmission in the power virtual private network equipment in time slot t; represents the access network delay, x m,k represents the connection relationship between service terminal m and base station k. m,k =1 indicates connection, D m Indicates the packet size, r m,k represents the upload data rate of service m at base station k; Indicates the core network delay, represents the node set of business m, n represents the node, f(D m ) indicates that the size of the transmitted data is D m The amount of computing resources required, C m,n (t) represents the amount of computing resources allocated to the transmission service m on node n, Indicates the corresponding link set, l indicates the link, d m,l represents the distance of link l, B m,l (t) represents the amount of bandwidth resources allocated to link l; f D (t) represents the delay index evaluation function, D represents the fixed delay; M t represents the total number of services in time slot t, E D (t) represents the delay indicator.

[0048] From the above description, it can be seen that the access network delay and the core network delay are taken into account in the process of calculating the delay, and the access network delay is described by the data packet size and the relationship between the service terminal and the base station, and the core network delay is described by data such as the amount of computing resources required to transmit data, the amount of computing resources allocated for transmission services, and the link distance, thereby more accurately reflecting the delay of the service.

[0049] Further, calculating the throughput index of the target network slice includes:

[0050]

[0051] Among them, f T (t) represents the throughput indicator evaluation function, r(t) represents the actual throughput of the deterministic service, is the minimum uplink transmission data rate of the slice, that is, the minimum throughput required to transmit service m; E T (t) represents the throughput index, M t Represents the total number of services in time slot t.

[0052] It can be seen from the above description that by obtaining the actual throughput of the deterministic service and comparing it with the minimum throughput required for transmitting the service, it is possible to effectively determine whether the actual throughput meets the service transmission requirements.

[0053] Furthermore, constructing a resource allocation optimization model according to the information freshness index, reliability index, latency index, and throughput index of the target network slice includes:

[0054] Q m =β1E D (t)+β2E T (t)+β3E A (t)+β4E R (t);

[0055] Among them, Q m represents the performance of a single network slice m; E D (t) represents the delay index; E T (t) represents the throughput index, E A (t) represents the information freshness index, E R (t) represents the reliability index; β1-β4 represent the weight values ​​corresponding to the index respectively.

[0056] From the above description, it can be seen that by setting corresponding weight values ​​for different indicators, the weight values ​​corresponding to each indicator can be adjusted according to actual business needs, so as to better evaluate the performance of the target network slice.

[0057] Furthermore, the optimizing and solving the resource allocation optimization model by using a deep reinforcement learning algorithm and a Markov decision process model includes:

[0058] Modeling the resource allocation optimization model as a Markov decision process, wherein the decision process includes a reward function;

[0059] Obtaining a resource allocation result based on the current state of the target network slice and historical experience through the deep reinforcement learning algorithm;

[0060] Determine whether the resource allocation result meets the business requirements, and if not, perform a negative penalty in the reward function;

[0061] The deep reinforcement learning algorithm is iterated repeatedly, and in each iteration round, the resource allocation result is adjusted according to the reward function until the business requirements are met.

[0062] From the above description, it can be seen that by designing a reward mechanism, the deep reinforcement learning algorithm can continuously improve its strategy to maximize the performance evaluation of the target network slice.

[0063] Furthermore, the deep reinforcement learning algorithm includes a model-free deep reinforcement learning algorithm based on maximum entropy.

[0064] From the above description, we can see that a model-free deep reinforcement learning algorithm based on maximum entropy is used to solve the problem. The optimal strategy of this algorithm is to maximize its entropy regularization reward, making the algorithm more stable.

[0065] Another embodiment of the present invention provides an end-to-end deterministic slice orchestration device based on deep reinforcement learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step in the above-mentioned end-to-end deterministic slice orchestration method based on deep reinforcement learning is implemented.

[0066] The end-to-end deterministic slice orchestration method and device based on deep reinforcement learning provided by the present invention can be applied to the scenario of network resource allocation in the power grid. By dynamically optimizing the allocation of network resources, the deterministic requirements of the power business on service quality such as latency and reliability can be met. The following is an explanation through specific implementation methods:

[0067] Embodiment 1

[0068] Please refer to Figure 1 , an end-to-end deterministic slice orchestration method based on deep reinforcement learning, comprising:

[0069] S1. Use virtualization technology to map physical network nodes to virtual networks to obtain an end-to-end network slice virtualization model. For example, taking 5G network slice deployment as an example, slices in different business scenarios have different resource requirements for the underlying physical network, and network slices also have differences in deployment structure. Specifically:

[0070] Please refer to Figure 2 , network slices of different service types are deployed on the underlying physical network composed of access network, transmission network and core network, and the slices are interconnected and collaborated with each other. Among them, the architecture of end-to-end deterministic network slicing mainly includes the physical perception layer, 5G power private network slicing layer and application service layer as well as the interfaces between the layers. The physical perception layer includes power business terminals, base stations, transmission equipment (such as routers, switches) and servers, which is an important basic guarantee for 5G power virtual private network slicing to achieve deterministic transmission; the 5G power private network slicing layer is responsible for virtual mapping of physical network nodes and links, and performs network resource scheduling and management. The application service layer includes slice business management and control and other types of applications.

[0071] Network slices include access network slices and core network slices. Access network slices are used to process the transmission from service terminals to base stations, and core network slices are used to manage the transmission links and computing resource allocation of services. The underlying physical network and the mapped virtual network span the access network domain and the core network domain. When a network slice is requested for deployment, each node corresponds to a node on the underlying physical network, and each link corresponds to a physical link connecting the node location.

[0072] Each node has a computing resource value attribute, and each communication link has a bandwidth resource value attribute. The physical infrastructure in the 5G power virtual private network is represented as an undirected graph G=(G N ,G L ), where G N ={1,2,…,n,…,N} represents a node set, G L ={1,2,…,l,…,L} represents the link set. The virtual undirected graph in the network slice is represented as in represents the node set of the virtual slice network, represents the link set of the virtual slice network; the computing resource set on the node is represented as C = {C1, C2, ..., C n ,…,C N}, the physical nodes are connected by wired links, and the bandwidth resources on the links are expressed as B = {B1, B2, ... B l ,…B L}.

[0073] For example, the access network is composed of k different cellular cells, represented by the set K = {1, 2, ..., k}. Each cell consists of a base station located at the center of the cell and m randomly distributed power service terminals. The power terminal service set is represented by M = {1, 2, ..., m}. Among them, whether the service terminal m is connected to the base station k is represented by a binary variable x m,k It means that when x m,k =1, service terminal m accesses cell k. Assuming that each service terminal m can only request one service at a time, terminal m can only be connected to one base station, which is expressed as:

[0074]

[0075] In the access network, the total bandwidth B Hz is divided into a group of physical resource blocks (PRBs). RB ={1,2,…,r} can only be dispatched to one power business terminal m, using binary variables indicates; when When , it means that the electric power service terminal m uses the spectrum resource block r within the cell, and the number of RB resources required by service m is:

[0076]

[0077] The RB resources allocated to all services on base station k cannot exceed the maximum RB limit of time-frequency resources:

[0078]

[0079] The upload data rate of power service m at base station k in time slot t is expressed as:

[0080]

[0081] Where p represents the transmit power, g represents the channel gain, represents the path loss from power terminal m to base station k, ω represents the path loss coefficient, σ 2 represents the additive white Gaussian noise power.

[0082] To ensure the QoS of the service, the uplink data rate should meet the minimum uplink transmission data rate of the slice:

[0083]

[0084] in, Indicates the minimum uplink transmission data rate required by service m. Different service requirements have different minimum uplink transmission data rates, and the network slice controller obtains the value in advance through service intent input.

[0085] In the core network, define a binary variable Indicates whether the underlying service node i used to transmit service m is mapped to virtual node u. If so, then otherwise Using binary variables Indicates whether the physical link (i, j) is mapped to the virtual link (u, v). If so, otherwise The total available computing resources on each node is C max , the computing resource constraints on virtual node u are met Among them C m,u (t) represents the computing resources allocated to service m on virtual node u in time slot t. The computing resource set C on the node = {C1, C2, …, C n ,…,C N} represents the computing resource set on N physical nodes. The maximum available bandwidth resource on each link is B max , satisfying the link bandwidth resource constraints Among them, B m,(u,v) (t) represents the bandwidth resources allocated to service m on the virtual link (u, v) in time slot t. The bandwidth resources on the link are B = {B1, B2, ... B l ,…B L} represents the bandwidth resource set on L physical links; let the tuple (C t ,B t ) represents the resource allocation set in time slot t, and represents the computing resources and bandwidth resources allocated to the network slice in time slot t. Different types of power services have different requirements for QoS indicators. Deterministic transmission needs to ensure that the total delay of the network slice is within a given range.

[0086] S2. Obtain resource data of the target network slice, and calculate the information freshness (AoI, Age of Information) index, reliability index, latency index and throughput index of the target network slice based on the resource data; that is, based on the service requirements of the power business, a resource allocation optimization model including information freshness and reliability requirements is proposed, aiming to maximize the long-term average satisfaction of the network slice; after obtaining the parameter value of the target parameter to be calculated and the preset comparison value corresponding to the target parameter, the evaluation value of the target parameter is obtained according to the ratio of the parameter value to the preset comparison value, and then the evaluation values ​​of all target parameters under the target network slice are obtained and the average value is calculated to obtain the target index corresponding to the target parameter, specifically:

[0087] (1) AoI refers to the time difference between the current moment and the last time a message was received. Specifically, for a certain information system, the smaller the AoI value, the newer the information and the more timely the transmission; conversely, the larger the AoI value, the older the information and the worse the timeliness. Information freshness is used to measure the real-time nature of power business data. By monitoring AoI, network resources are dynamically adjusted to improve the timeliness of business transmission. AoI can be used as a key indicator to measure network QoS. By evaluating AoI, the allocation of network resources can be optimized and the overall service quality can be improved. By measuring information freshness through AoI, the actual AoI of deterministic business is defined as:

[0088]

[0089] Among them, a m (t+1) represents the information freshness of business m at time t+1; binary variable Indicates that the power virtual private network device chooses to successfully transmit data in time slot t. If so, then otherwise, d(t) represents the end-to-end delay of data transmission in the power virtual private network device in time slot t. If the power virtual private network device successfully receives data during time slot t, the AoI is reset to d(t), otherwise the AoI increases by a0; a0 is a fixed constant preset by the system;

[0090] The average AoI of the power deterministic service m is expressed as:

[0091]

[0092] For services with a large number of control instructions and high information timeliness requirements in 5G power virtual private networks, the AoI index is not allowed to exceed the specified threshold, otherwise, the delay of control instructions will cause unpredictable consequences. Therefore, AoI is an important indicator for deterministic services. For deterministic service processing, when the information age value is greater than the previous requirement value of the service, the network transmission of the service is considered invalid transmission. The service information age indicator evaluation function is defined as:

[0093]

[0094] a(t) represents the timeliness of deterministic services, A m represents the preset timeliness index of the transmission service m. On this basis, the information age evaluation function is defined to describe the satisfaction of the deterministic service with the transmission:

[0095]

[0096] M t represents the total number of services in time slot t, E A (t) represents the information freshness index.

[0097] (2) Reliability is essential for consistent performance and uninterrupted service, and is also a key aspect to ensure the determinism of the entire network. Due to damage to software and hardware or the influence of the external environment, service nodes may experience hardware or software failures (for example, unexpected restart / shutdown of physical nodes, network disconnection, software errors, etc.), which may cause interruptions and lead to failure of power grid service transmission tasks. The reliability model evaluates the reliability of power service transmission links by calculating the working probability of virtual nodes and physical nodes to ensure service continuity during network interruptions or failures. Software reliability is defined as the probability that a VNF successfully completes its expected service processing. Software reliability depends on the complexity, configuration errors, and software errors of the VNF. The probability of each VNF node u working normally is: p soft,u ∈(0,1); Hardware reliability is related to the equipment quality of the physical nodes that carry the VNF. The probability that each physical node i works normally is: p phy,i ∈(0,1);

[0098] The power service m transmits data through a series of virtual links connected by VNFs. The failure probability of each service transmission data failure and server node failure is defined separately. The reliability of service m can be calculated as the product of the reliabilities of multiple nodes in the link, and a reliability model is established:

[0099]

[0100] Among them, R m (t) represents the reliability of service m at time t; since the status of VNF or physical node may change, such as failure or repair, the reliability model will adjust the probability according to the time change, reflecting the dynamic reliability of the node or link; u virtual node; The node set representing the service m; the characteristic function of the service transmission reliability index is defined as:

[0101]

[0102] Among them, R(t) represents the reliability of deterministic services, R m represents the preset reliability index of the transmission service m; on this basis, the reliability evaluation function is defined as:

[0103]

[0104] f R (t) represents the reliability index evaluation function, E R (t) represents the reliability index.

[0105] (3) The end-to-end latency of the 5G power virtual private network is the latency of all network links and devices from the time the service request is sent from the service request terminal to the time the service end processes it. For end-to-end network slice requests, the user's latency includes the access network latency and the core network latency, so the latency is defined as:

[0106]

[0107] The delay from the power equipment terminal m to the base station k is the access network delay, which is expressed as:

[0108]

[0109] The core network latency is mainly composed of computing latency and link latency. The computing latency refers to the time from when the service data is received and calculated by the processing node to when it is forwarded to the link. The link latency refers to the time it takes for the data to be transmitted on the link. Therefore, the core network latency is expressed as:

[0110]

[0111] Where d(t) represents the end-to-end delay of data transmission in the power virtual private network equipment in time slot t; represents the access network delay, x m,k represents the connection relationship between service terminal m and base station k. m,k =1 indicates connection, D m Indicates the packet size (in bits), r m,k represents the upload data rate of service m at base station k; Indicates the core network delay, represents the node set of business m, n represents the node, f(D m ) indicates that the size of the transmitted data is D m The amount of computing resources required, C m,n (t) represents the amount of computing resources allocated to the transmission service m on node n, Indicates the corresponding link set, l indicates the link, d m,l represents the distance of link l, B m,l (t) represents the amount of bandwidth resources allocated to link l;

[0112] Slice resource scheduling is guided by the mapping relationship between performance indicators and resources, and the characteristic function of transmission delay is defined as:

[0113]

[0114] The end-to-end delay evaluation function of the service within time t is defined as:

[0115]

[0116] f D (t) represents the delay index evaluation function, D represents the fixed delay; E D (t) represents the delay indicator.

[0117] (4) The characteristic function of the service transmission throughput indicator is defined as:

[0118]

[0119] Among them, f T (t) represents the throughput indicator evaluation function, r(t) represents the actual throughput of the deterministic service, is the minimum uplink transmission data rate of the slice, that is, the minimum throughput required for transmitting service m; based on this, the transmission throughput evaluation function of the time slot t service is defined as:

[0120]

[0121] E T (t) represents the throughput indicator.

[0122] S3. Build a resource allocation optimization model based on the information freshness index, reliability index, latency index, and throughput index of the target network slice. Specifically:

[0123] Each end-to-end network slice considers four key indicators: latency D, throughput r, timeliness A, and reliability R; the value range of each element in service satisfaction is {E D (t),E T (t),E A (t),E R (t)}∈[0,1], the closer it is to 1, the higher the service satisfaction; Based on the above analysis, in order to measure the QoS of the system, the performance evaluation function Q of the network slice is used m It indicates the evaluation criteria for the network to complete transmission according to specific deterministic business indicators within a given transmission cycle; Q m Only the performance of a single network slice m is shown:

[0124] Q m =β1E D (t)+β2E T (t)+β3E A (t)+β4E R (t);

[0125] β1-β4 represent the weight values ​​corresponding to the indicators respectively;

[0126] According to Q mDetermine the transmission status of deterministic services and guide link resource scheduling; the optimization goal is to maximize the performance evaluation function of the network slice, and the optimization model is shown in the formula:

[0127]

[0128] C6:β1+β2+β3+β4=1.

[0129] S4. The resource allocation optimization model is optimized and solved through the deep reinforcement learning algorithm and the Markov decision process model to obtain the resource allocation plan of the target network slice. Specifically:

[0130] S41. Model the resource allocation optimization model as a Markov decision process (MDP), including a state set, an action set, a reward function, and a state transition matrix. For example, the MDP consists of a five-tuple Composition, of which represents a finite set of states, represents a finite set of actions, is the state transfer matrix, is the reward function, and γ represents the discount factor. The end-to-end network slice controller observes the current state as an intelligent agent. The state set observed at time slot t is: S(t) = {ω s (t),C s (t),B s (t),s(t)}; where ω s (t) represents the available RB resources observed by the base station at time t, C s (t),B s (t) are the remaining computing resources of the node and the remaining bandwidth resources of the link in time slot t; Determine the maximum latency requirement, maximum data freshness requirement, minimum rate, and minimum reliability requirement for reaching the service. The slice controller selects actions based on the observed state. The action set is: Among them, ω(t), C(t), and B(t) represent the RB resources, computing resources, and bandwidth resources allocated by nodes and links to each service, respectively.

[0131] S42. Obtain resource allocation results based on the current state of the target network slice and historical experience through a deep reinforcement learning algorithm; for example, the network slice controller selects the optimal resource allocation solution based on the current state and historical experience, and the selected action determines the allocation of computing resources and bandwidth resources; the ultimate goal of the network slice controller is to maximize the performance evaluation function under the constraints of the time slot system; after taking the resource allocation action, the network slice controller can immediately obtain a reward, and by designing a reward mechanism, the intelligent agent can continuously improve its strategy to maximize the performance evaluation function; the solution is valid only when all constraints are met, otherwise the optimization problem needs to be re-solved to obtain a new solution.

[0132] S43. Determine whether the resource allocation result meets the service requirements. If not, a negative penalty is imposed in the reward function. That is, a penalty mechanism is introduced into the reward function. If the action of the network slice controller does not meet the QoS requirements of the service, a negative penalty is imposed in the reward to guide the network slice controller to perform better resource allocation.

[0133] Define the reward as:

[0134]

[0135] The penalty term is a negative constant. If the action taken does not satisfy the constraints, a penalty is added to the reward.

[0136] S44. The deep reinforcement learning algorithm is iterated repeatedly, and in each iteration round, the resource allocation result is adjusted according to the reward function until the business requirements are met. That is, the network slice controller gradually improves its strategy through repeated iterations. In each iteration round, the network slice controller adjusts its strategy based on feedback to maximize the cumulative reward and ensure efficient use of resources. In addition, the multi-agent system in deep reinforcement learning can optimize the efficiency of network resource use through the collaborative work of multiple agents, and adaptively adjust when the network load changes to ensure the normal operation of key services.

[0137] Please refer to Figure 3 Based on the end-to-end deterministic network service slicing architecture of 5G power virtual private network, a deterministic business resource allocation method based on flexible actor-critic (SAC) is proposed; SAC is an offline policy algorithm based on continuous action space and oriented to the maximum entropy RL framework. The optimal strategy of this method is to maximize its entropy regularized reward, making the algorithm more stable; In addition to learning the strategy to maximize the expected cumulative reward, the objective function of the SAC algorithm also requires that the action entropy output of each strategy is maximized:

[0138]

[0139] In order to automatically update the temperature coefficient α, the optimized loss function is:

[0140]

[0141] The value function and strategy of SAC are each fitted by a neural network controlled by parameters. The actor network is updated according to the minimum KL divergence. Two Q networks are used and the smaller Q value is selected as the target Q value each time to ensure fast and stable training. The critic network is updated by minimizing the Bellman error. The end-to-end slice deterministic service resource allocation algorithm specifically includes the following steps:

[0142] 1. Initialize the system's slice controller training environment and configure the SAC network model in deep reinforcement learning. The network model includes the main network and the target network, and sets the experience replay pool D;

[0143] 2. Start the iterative optimization process, set the number of iterations M, initialize the environmental state of the slice controller, and start observing the current network state, and execute step S3;

[0144] 3. At each time step t, the slice controller selects an action through a greedy strategy, executes the resource allocation action, calculates the feedback reward and enters the next state;

[0145] 4. Experience (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool D, and experience data is randomly sampled from the experience replay pool to train the strategy network;

[0146] 5. Calculate the time difference target The loss function is minimized by the gradient descent method, and then the SAC main network parameters are updated and copied to the target network parameters.

[0147] S5. Allocate resources to different businesses according to the resource allocation plan.

[0148] Embodiment 2

[0149] Please refer to Figure 4 , an end-to-end deterministic slice orchestration device based on deep reinforcement learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in an end-to-end deterministic slice orchestration method based on deep reinforcement learning as described in Example 1 is implemented.

[0150] In summary, the present invention provides an end-to-end deterministic slice orchestration method and device based on deep reinforcement learning. The present invention ensures that the deterministic needs of the power business are met by introducing an optimization model including key indicators such as information freshness and reliability; an end-to-end deterministic slice orchestration method based on deep reinforcement learning in a 5G power virtual private network is proposed. Compared with the traditional fixed resource allocation strategy, this method can maintain a high level of business satisfaction under different network conditions through dynamic resource allocation and intelligent scheduling, ensuring that the power business obtains a stable and efficient service experience in the 5G virtual private network, ensuring the stable operation of the power deterministic network business, and improving resource utilization.

[0151] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An end-to-end deterministic slice orchestration method based on deep reinforcement learning, characterized in that: include: Map physical network nodes to virtual networks through virtualization technology to obtain an end-to-end network slicing virtualization model; Obtain resource data of the target network slice, and calculate the information freshness index, reliability index, latency index, and throughput index of the target network slice according to the resource data; Building a resource allocation optimization model according to the information freshness index, reliability index, latency index and throughput index of the target network slice; The resource allocation optimization model is optimized and solved by a deep reinforcement learning algorithm and a Markov decision process model to obtain a resource allocation solution for the target network slice; Resources are allocated to different services according to the resource allocation scheme.

2. The end-to-end deterministic slice orchestration method based on deep reinforcement learning according to claim 1, characterized in that: The calculating the information freshness index, reliability index, latency index and throughput index of the target network slice according to the resource data includes: Obtaining a parameter value of a target parameter to be calculated and a preset comparison value corresponding to the target parameter; The target parameters include the information freshness, reliability, latency and throughput; Obtaining an evaluation value of the target parameter according to a ratio of the parameter value to the preset comparison value; Obtain the evaluation values ​​of all the target parameters under the target network slice and calculate the average value to obtain the target indicator corresponding to the target parameter.

3. The end-to-end deterministic slice orchestration method based on deep reinforcement learning according to claim 2 is characterized in that: Calculating the information freshness index of the target network slice includes: Among them, a m (t+1) represents the information freshness of business m at time t+1; binary variable Indicates that the power virtual private network device chooses to successfully transmit data in time slot t. If so, then otherwise, d(t) represents the end-to-end delay of data transmission in time slot t of the power virtual private network equipment; a0 is a fixed constant preset by the system; a(t) represents the average information freshness; f A (t) represents the information freshness index evaluation function, A m represents the preset timeliness index of the transmission service m; M t represents the total number of services in time slot t, E A (t) represents the information freshness index.

4. The end-to-end deterministic slice orchestration method based on deep reinforcement learning according to claim 2 is characterized in that: Calculating the reliability index of the target network slice includes: p soft,u ∈(0,1)； p phy,i ∈(0,1)； Among them, R m (t) represents the reliability of service m at time t; u is a virtual node; A set of nodes representing business m; binary variable Indicates whether the underlying service node i used to transmit service m is mapped to virtual node u. If so, then otherwise p soft,u ∈(0,1) represents the probability that software node u works normally; p phy,i ∈(0,1) represents the probability that physical node i works normally; f R (t) represents the reliability index evaluation function, R m Represents the preset reliability index of the transmission service m; M t represents the total number of services in time slot t, E R (t) represents the reliability index.

5. The end-to-end deterministic slice orchestration method based on deep reinforcement learning according to claim 2, characterized in that: Calculating the latency index of the target network slice includes: Where d(t) represents the end-to-end delay of data transmission in the power virtual private network equipment in time slot t; represents the access network delay, x m,k represents the connection relationship between service terminal m and base station k. m,k =1 indicates connection, D m Indicates the packet size, r m,k represents the upload data rate of service m at base station k; Indicates the core network delay, represents the node set of business m, n represents the node, f(D m ) indicates that the size of the transmitted data is D m The amount of computing resources required, C m,n (t) represents the amount of computing resources allocated to the transmission service m on node n, Indicates the corresponding link set, l indicates the link, d m,l represents the distance of link l, B m,l (t) represents the amount of bandwidth resources allocated to link l; f D (t) represents the delay index evaluation function, D represents the fixed delay; M t represents the total number of services in time slot t, E D (t) represents the delay indicator.

6. The end-to-end deterministic slice orchestration method based on deep reinforcement learning according to claim 2, characterized in that: Calculating the throughput index of the target network slice includes: Among them, f T (t) represents the throughput indicator evaluation function, r(t) represents the actual throughput of the deterministic service, is the minimum uplink transmission data rate of the slice, that is, the minimum throughput required to transmit service m; E T (t) represents the throughput index, M t Represents the total number of services in time slot t.

7. The end-to-end deterministic slice orchestration method based on deep reinforcement learning according to claim 1, characterized in that: The constructing of a resource allocation optimization model according to the information freshness index, reliability index, latency index, and throughput index of the target network slice includes: Q m =β1E D (t)+β2E T (t)+β3E A (t)+β4E R (t); Among them, Q m represents the performance of a single network slice m; E D (t) represents the delay index; E T (t) represents the throughput index, E A (t) represents the information freshness index, E R (t) represents the reliability index; β1-β4 represent the weight values ​​corresponding to the index respectively.

8. The end-to-end deterministic slice orchestration method based on deep reinforcement learning according to claim 1, characterized in that: The optimizing and solving the resource allocation optimization model by using a deep reinforcement learning algorithm and a Markov decision process model includes: Modeling the resource allocation optimization model as a Markov decision process, wherein the decision process includes a reward function; Obtaining a resource allocation result based on the current state of the target network slice and historical experience through the deep reinforcement learning algorithm; Determine whether the resource allocation result meets the business requirements, and if not, perform a negative penalty in the reward function; The deep reinforcement learning algorithm is iterated repeatedly, and in each iteration round, the resource allocation result is adjusted according to the reward function until the business requirements are met.

9. The end-to-end deterministic slice orchestration method based on deep reinforcement learning according to claim 7, characterized in that: The deep reinforcement learning algorithm includes a maximum entropy-based model-free deep reinforcement learning algorithm.

10. An end-to-end deterministic slice orchestration device based on deep reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps in an end-to-end deterministic slice orchestration method based on deep reinforcement learning as described in any one of claims 1-9.

Images