Optimization method and device for OTN network resources, computer device and medium
By implementing global optimization of OTN network resources, the problem of uneven resource allocation in existing technologies is solved, and the efficiency of network resource utilization and the ordering strategy for service creation are improved.
Patent Information
- Application Number
- CN202010899413.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-31
- Publication Date
- 2025-12-30
- Estimated Expiration
- 2040-08-31
AI Technical Summary
Existing technologies struggle to optimize resource allocation in OTN networks, resulting in high operation and maintenance costs, poor transmission performance, and an inability to meet users' overall resource allocation optimization goals for network services.
By employing reinforcement learning algorithms, the system determines the services to be built through action policies, calculates timely rewards, iteratively updates and optimizes the target policy parameters, optimizes the order of OTN network service creation, and achieves global resource optimization.
It improves the economic efficiency of OTN network operation and maintenance, reduces operation and maintenance costs, improves transmission performance and quality, and optimizes OTN network resources.
Smart Images
Figure CN114125595B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of automatic control technology, and specifically to an OTN network resource optimization method, apparatus, computer equipment, and computer-readable medium. Background Technology
[0002] With the development of artificial intelligence technology, reinforcement learning technology is receiving increasing attention across various fields and industries. Reinforcement learning, also known as reward learning or evaluation learning, is an important machine learning method with numerous applications in areas such as intelligent robot control and network analysis and prediction. Within the connectionist machine learning school, learning algorithms are categorized into three types: unsupervised learning, supervised learning, and reinforcement learning.
[0003] Reinforcement learning is a learning process where an agent learns through trial and error. Rewards gained through interaction with the environment guide behavior, with the goal of maximizing the agent's reward. Reinforcement learning differs from supervised learning in connectionist learning, primarily in the reinforcement signal. In reinforcement learning, the reinforcement signal provided by the environment evaluates the quality of the action (usually a scalar signal), rather than instructing the reinforcement learning system (RLS) on how to produce the correct action. Because the external environment provides limited information, RLS must learn through its own experience. In this way, RLS acquires knowledge in the action-evaluation environment and improves its action plans to adapt to the environment.
[0004] In recent years, with the application and promotion of reinforcement learning technology, how to apply the advantages of this technology to the field of intelligent management and operation of OTN (Optical Transport Network), especially the application of reinforcement learning in OTN network resource optimization, has received widespread attention from experts in the OTN field.
[0005] A global co-current optimization (GCO) solution based on the SDON (Software Defined Optical Network) architecture, such as... Figure 1As shown, the main purpose of GCO (Global Resource Optimization) is to maximize the fulfillment of the user's (network service operator's) predetermined resource allocation goals for the overall network services during the planning or batch creation of OTN network services, while avoiding obstruction of routing calculations and resource allocation for each service. This requires that the calculated routes and total resource usage for each OTN service maximize the fulfillment of the user's (network service operator's) predetermined resource allocation optimization goals for the entire network service. Through network-wide resource optimization technology, the user's CAPEX (capital expenditure) / OPEX (operating expenses) can be reduced to the greatest extent possible, while increasing operational revenue and optimizing transmission performance and quality. This is directly related to the economic benefits of the user's network operation; therefore, this technology has received high attention from users, and how to achieve OTN network resource optimization is of great significance. Summary of the Invention
[0006] This disclosure addresses the aforementioned deficiencies in the prior art by providing an OTN network resource optimization method, apparatus, computer device, and computer-readable medium.
[0007] In a first aspect, embodiments of this disclosure provide an OTN network resource optimization method, including:
[0008] Based on the action strategy, determine the business to be built in the current business establishment state, create the business to be built, calculate the timely reward in the current business establishment state, enter the next business establishment state, until the end of a round, calculate and update the optimization target strategy parameters in each business establishment state based on the timely reward in each business establishment state.
[0009] The process iterates a preset number of times to calculate and update the optimization target strategy parameters for each business establishment state.
[0010] Based on the optimization target strategy parameters of each service establishment state in the preset number of rounds, determine the optimal optimization target strategy parameters for each service establishment state.
[0011] The action strategy is updated based on the optimal target strategy parameters of each business establishment state.
[0012] In some embodiments, the step of calculating and updating the optimization target strategy parameters for each service establishment state based on the timely reward for each service establishment state includes:
[0013] The expected return for the current business establishment status is calculated based on the timely rewards for each business establishment status after the next business establishment status.
[0014] The optimization target strategy parameters are calculated and updated based on the expected return under the current business establishment status.
[0015] In some embodiments, the expected return under the current business establishment status is calculated according to the following formula:
[0016]
[0017] Among them, G t Establish state S for business t Next, execute action a t The expected return, γ is the discount factor, 0 < γ < 1; R is the immediate reward, and t is the business establishment state S. t The number of services already created, t = (0, ..., n-1), where n is the total number of services to be created in the OTN network.
[0018] In some embodiments, the expected return in the last business establishment state of each round is the actual total return of the round. After calculating the actual total return in the last business establishment state of each round, and before calculating the optimization target strategy parameters in the last business establishment state of each round, the method further includes:
[0019] The actual total return is updated based on the actual total return, a preset threshold, and a preset additional return; wherein, if the actual total return is greater than or equal to the threshold, then the actual total return = actual total return + the additional return; if the actual total return is less than the threshold, then the actual total return = actual total return - the additional return.
[0020] In some embodiments, the optimization target policy parameter includes the state behavior value Q. π (s,a), Alternatively, the optimization objective strategy parameters include the state value V. π (s), Where π(a|s) is the probability of taking action a according to action strategy π(s,a) under the service establishment state S, and A is the set of actions executed under each service establishment state.
[0021] In some embodiments, when the optimization target policy parameter is the state behavior value Q π When (s,a), the Monte Carlo algorithm, the time difference algorithm with different strategies, or the time difference algorithm with the same strategy are used to calculate and update the optimization target strategy parameters in each service establishment state.
[0022] The step of updating the action strategy based on the optimal optimization target strategy parameters under each service establishment state includes: based on the state behavior value Q π (s,a) Update the action policy.
[0023] In some embodiments, when the optimization target policy parameter is the state value V πWhen (s), the dynamic programming algorithm is used to calculate the optimization objective strategy parameters;
[0024] The step of updating the action strategy based on the optimal optimization target strategy parameters under each service establishment state includes: based on the state value V π (s) Update the action policy.
[0025] In some embodiments, determining the pending service in the current service establishment state according to the action strategy includes:
[0026] Calculate the probability of selecting each pending business under the current business establishment status;
[0027] Based on the probability of selecting each pending service under the current service establishment status, a pending service is determined;
[0028] Based on the preset OTN network optimization objective function, the candidate routes for the identified services to be built are sorted.
[0029] Based on the number of candidate routes in the sorting, calculate the selection probability of each candidate route in the sorting.
[0030] Based on the selection probability of each candidate route in the sorting, a candidate route is determined as the route for the service to be established in the current service establishment state.
[0031] In some embodiments, the probability of selecting each pending service in the current service establishment state is calculated according to the following formula: Where p(s) tsvci Establish state S for the business t Select the i-th service Svc i The probability of establishing state S in the business is t. t The number of services already created, t = (0, ..., n-1), where n is the total number of services to be created in the OTN network.
[0032] In some embodiments, the objective function for OTN network optimization includes minimizing the routing cost of OTN network services or minimizing the latency of OTN network services.
[0033] In some embodiments, calculating the selection probability of each candidate route in the sorting based on the number of candidate routes in the sorting includes:
[0034] If the number of candidate routes in the sorting is m = 1, then the probability of selecting a candidate route is:
[0035] If the number of candidate routes in the sorting is m = 2, then the probability of selecting one of the candidate routes is: The probability of selecting the other alternative route is:
[0036] If the number of candidate routes m in the sorting is greater than 2, then the selection probability of candidate routes 1 to m-1 is: The probability of selecting the m-th alternative route is:
[0037] Where ε is the greed coefficient, 0.5 < ε < 1, and t is the business establishment state S. t The number of services already created is t = (0, ..., n-1), where n is the total number of services to be created in the OTN network, and i = (1, ..., m-1).
[0038] In another aspect, embodiments of this disclosure also provide an OTN network resource optimization device, including: a first processing module, a second processing module, and an update module.
[0039] The first processing module is used to determine the business to be built in the current business establishment state according to the action strategy, create the business to be built, calculate the timely reward in the current business establishment state, enter the next business establishment state, until the end of a round, calculate and update the optimization target strategy parameters in each business establishment state according to the timely reward in each business establishment state; iterate for a preset number of rounds to calculate and update the optimization target strategy parameters in each business establishment state.
[0040] The second processing module is used to determine the optimal optimization target strategy parameters for each service establishment state based on the optimization target strategy parameters for each service establishment state in the preset number of rounds.
[0041] The update module is used to update the action strategy according to the optimal optimization target strategy parameters of each service establishment state.
[0042] In another aspect, embodiments of this disclosure also provide a computer device, including:
[0043] One or more processors;
[0044] A storage device on which one or more programs are stored;
[0045] When the one or more programs are executed by the one or more processors, the one or more processors implement the OTN network resource optimization method as described above.
[0046] In another aspect, embodiments of this disclosure also provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed, implements the OTN network resource optimization method as described above.
[0047] The OTN network resource optimization method and apparatus provided in this disclosure include: determining the services to be built in the current service establishment state according to an action policy; creating the services to be built; calculating the timely reward in the current service establishment state; entering the next service establishment state until the end of a round; calculating and updating the optimization target policy parameters in each service establishment state according to the timely reward in each service establishment state; iterating for a preset number of rounds to calculate and update the optimization target policy parameters in each service establishment state; determining the optimal optimization target policy parameters in each service establishment state according to the optimization target policy parameters in each service establishment state in the preset number of rounds; and updating the action policy according to the optimal optimization target policy parameters in each service establishment state. This disclosure utilizes the reward and punishment mechanism of a reinforcement learning algorithm to optimize the order of OTN network service creation, resulting in action policies with good convergence, rigor, and high reliability. It reduces the OTN network resource optimization problem to a problem of ordering OTN network service creation, and achieves global optimization of OTN network resources by obtaining optimized action policies. Attached Figure Description
[0048] Figure 1 This is a schematic diagram of network-wide resource optimization under the SDON architecture provided in this embodiment of the disclosure;
[0049] Figure 2 This is a schematic diagram of the OTN network resource optimization process provided in the embodiments of this disclosure;
[0050] Figure 3 A flowchart illustrating the calculation of target strategy parameters provided in this embodiment of the disclosure;
[0051] Figure 4 A schematic diagram illustrating the process for determining a pending service in the current service establishment state, provided in an embodiment of this disclosure;
[0052] Figure 5 A schematic diagram of the structure of the OTN network resource optimization device provided in the embodiments of this disclosure. Detailed Implementation
[0053] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, these exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this disclosure.
[0054] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.
[0055] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the said feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded.
[0056] The embodiments described herein can be described with reference to plan views and / or cross-sectional views using the ideal schematic diagrams of this disclosure. Therefore, the example illustrations can be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to those shown in the drawings, but include modifications to configurations formed based on manufacturing processes. Therefore, the areas illustrated in the drawings are schematic in nature, and the shapes of the areas shown in the figures illustrate specific shapes of areas of an element, but are not intended to be limiting.
[0057] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.
[0058] In the deployment of existing OTN network services, it is typically necessary to allocate corresponding OTN network resources (such as bandwidth, spectrum, wavelength, modulation format, routing, etc.) to each service according to operational needs. This requires optimizing the resources of the entire service under specified optimization strategies, including minimizing overall service latency and routing costs. Simultaneously, considering maximizing operational revenue, optimizing service performance, and minimizing CAPEX / OPEX, OTN service operation also needs to adhere to the established optimization strategies to achieve overall optimization in the allocation and use of network resources for OTN services. This includes minimizing OTN service latency, minimizing OTN network routing costs, or maximizing OTN network bandwidth utilization. This necessitates that during the creation process of OTN network services, both the service's own resource optimization and the orchestration of the creation order of all services must satisfy the global optimization of OTN network resource usage.
[0059] OTN network service creation typically employs a concurrent creation approach, meaning multiple services are created in batches at a single point in time. The service creation process essentially determines the creation order of all services, which in turn dictates the utilization and optimization of OTN network resources. This creation order is termed the service creation orchestration strategy (i.e., action strategy). A good service creation orchestration strategy can meet the optimization requirements of OTN network services for network resource usage.
[0060] During the initialization phase, n OTN services are created based on the environmental conditions of the OTN network topology (including mesh, star, and other structure types). The network environment state, action space, action optimization objective policy, and action policy are initialized. The relevant parameters of the reinforcement learning algorithm are defined as follows.
[0061] 1. Define the objective function for OTN network optimization.
[0062] The optimization objective function for OTN networks can be to minimize the routing cost of OTN network services. MinCost .
[0063] Among them, Cost(Svc i ) represents the i-th business Svc i The routing cost.
[0064] The objective function for OTN network optimization can also be to minimize the latency of OTN network services. MinDelay .
[0065] Among them, Delay(Svc i ) represents the i-th business Svc i The time delay.
[0066] 2. Define the feature vector of the business establishment state S.
[0067] The feature vector φ(s) is used to describe the service establishment status. The feature vector φ(s) is used to indicate which services have been created and which services have not yet been created. When a service to be established is completed, the next service establishment status is entered.
[0068] The feature vector φ(s) of the service establishment state S is described as follows:
[0069] φ(s)={StateID;SumSvcCost;SumSvcDelay;SvcNum;...SvcID i SvcCost i ;
[0070] SvcDelayi ;SvcSeqID i ;SvcRtID i ;SrcNdID i ;DstNdID i ...}
[0071] in,
[0072] StateID is used to establish a state ID for the business;
[0073] SumSvcCost is the sum of the routing costs for all current services; the routing cost for services not yet created is 0.
[0074] SumSvcDelay is the sum of the delays of all current services; the delay for services not yet created is 0.
[0075] SvcNum represents the total number of services in the OTN network, which is the sum of the number of existing services and the number of services to be built.
[0076] The following feature vector elements characterize a sequence of service establishment state attributes for the i-th service in the network. The ellipses before and after the ellipses indicate the service establishment state attribute sequences of the first i-1 services and the last ni services with the same definition.
[0077] SvcID i Let i be the business ID of the i-th business;
[0078] SvcCost i Let $\frac{i}{i}$ be the routing cost for the $i$-th service. If the service has not yet been created, the routing cost is 0.
[0079] SvcDelay i Let be the latency of the i-th service. If the service has not yet been created, the latency is 0.
[0080] SvcSeqID i Let be the sequence ID of the i-th service in the OTN network service. If the service has not yet been created, the sequence ID of the service is 0.
[0081] SvcRtID i This is the route ID used by the i-th service. If the service has not yet been created, the route ID for that service is 0.
[0082] SrcNdID i Let i be the ID of the i-th business source node;
[0083] DstNdID i Let be the node ID of the i-th business destination.
[0084] 3. Define the episode
[0085] An episode is defined as the sequential establishment of OTN network services using a specific action strategy.
[0086] 4. Define action a t and action strategies
[0087] An action refers to the process of selecting one service to be created from among the services to be created in the current network topology environment, selecting one of the multiple alternative routes (routes that have been allocated network resources) for the service to be created, and completing the creation of the service.
[0088] Action policy π(s,a) or μ(s,a) (also written as μ(s)) represents the creation order of services to be built (including the routes of the services to be built).
[0089] This disclosure provides an OTN network resource optimization method, such as... Figure 2 As shown, the method includes the following steps:
[0090] Step 11: Determine the pending business under the current business establishment state according to the action strategy, create the pending business, calculate the timely reward under the current business establishment state, and enter the next business establishment state until the end of a round. Calculate and update the optimization target strategy parameters under each business establishment state based on the timely reward under each business establishment state.
[0091] In this step, within a round, a service to be built is determined based on the action policy (including determining the route of the service to be built). After the service to be built is created, the immediate reward for the service establishment state is calculated. The current service establishment state ends, and the process moves to the next service establishment state. Following the above steps, for each service establishment state in a round, services to be built are created, and the immediate reward for the corresponding service establishment state is calculated, until the end of a round. Based on the immediate rewards for each service establishment state, the optimization target policy parameters for each service establishment state are calculated and updated.
[0092] In this step, different algorithms can be used to calculate and update the target policy parameters. It should be noted that different algorithms will optimize different target policy parameters. The various algorithms will be explained in detail later.
[0093] Step 12: Iterate for a preset number of rounds to calculate and update the optimization target strategy parameters for each service establishment state.
[0094] In this step, step 11 is repeated for a preset number of rounds, and the optimization target strategy parameters for each business establishment state in each round are calculated and updated.
[0095] Step 13: Determine the optimal optimization target strategy parameters for each service establishment state based on the optimization target strategy parameters for each service establishment state in the preset number of rounds.
[0096] In this step, for each service establishment state, the optimal optimization target policy parameters for that state are determined from the optimization target policy parameters from different rounds. It should be noted that the method for determining the optimal optimization target policy parameters varies depending on the algorithm used. Through this step, the optimal optimization target policy parameters for all service establishment states corresponding to all services to be established in the OTN network can be obtained.
[0097] Step 14: Update the action strategy according to the optimal optimization target strategy parameters of each service establishment state.
[0098] Optimize the target strategy parameters to characterize the business establishment state S t and action a t Once the optimal optimization objective strategy parameters for a certain service establishment state are determined, the optimal action 'a' for that service establishment state can be determined. t Optimal action a t This refers to the action of creating the optimal service to be built under the current service establishment state. This allows us to determine the optimal service to be built under the current service establishment state (including the route of the service to be built). From this, we can obtain the services to be built sorted according to the service establishment state. The sorting of the services to be built is the optimized action strategy.
[0099] The OTN network resource optimization method and apparatus provided in this disclosure include: determining the services to be built in the current service establishment state according to an action policy; creating the services to be built; calculating the timely reward in the current service establishment state; entering the next service establishment state until the end of a round; calculating and updating the optimization target policy parameters in each service establishment state according to the timely reward in each service establishment state; iterating for a preset number of rounds to calculate and update the optimization target policy parameters in each service establishment state; determining the optimal optimization target policy parameters in each service establishment state according to the optimization target policy parameters in each service establishment state in the preset number of rounds; and updating the action policy according to the optimal optimization target policy parameters in each service establishment state. This disclosure utilizes the reward and punishment mechanism of a reinforcement learning algorithm to optimize the order of OTN network service creation, resulting in action policies with good convergence, rigor, and high reliability. It reduces the OTN network resource optimization problem to a problem of ordering OTN network service creation, and achieves global optimization of OTN network resources by obtaining optimized action policies.
[0100] R t Indicates the service establishment status S t The timely reward obtained below, t is the business state S. t The number of services already created is given by t = (0, ..., n-1), where n is the total number of services to be created in the OTN network. It should be noted that timely rewards R... t The calculation method is related to the pre-set OTN network optimization objective function. Taking "minimizing the routing cost of OTN network services" as an example, R... t The value is the SvcID of the most recently created business under the current business establishment status. i Routing cost SvcCost i The reciprocal of R t =1 / SvcCost i In this way, services with lower routing costs are given state S. t The greater the immediate reward, the better, where R0 = 0 in state S0.
[0101] In some embodiments, such as Figure 3 As shown, the step of calculating and updating the optimization target strategy parameters for each business establishment state based on the timely rewards for each business establishment state includes the following steps:
[0102] Step 21: Calculate the expected return for the current business establishment state based on the timely rewards for each business establishment state after the next business establishment state.
[0103] In some embodiments, the expected return under the current business establishment status can be calculated according to the following formula:
[0104]
[0105] Among them, G t Establish state S for business t Next, execute action a t The expected return, γ is the discount factor, 0 < γ < 1; R is the immediate reward, and t is the business establishment state S. t The number of services already created, t = (0, ..., n-1), where n is the total number of services to be created in the OTN network.
[0106] It should be noted that the expected return in the last business establishment state is the same as the timely reward in that business establishment state.
[0107] Step 22: Calculate and update the optimization target strategy parameters in the current business establishment state based on the expected return in the current business establishment state.
[0108] Steps 21-22 utilize the reward and punishment mechanism of the reinforcement algorithm to optimize the parameters of the optimization target strategy.
[0109] In some embodiments, the expected return G in the last business establishment state in each round t The OTN network resource optimization method may further include the following steps, after calculating the actual total reward G for the last service establishment state in each round and before calculating the optimization target strategy parameters for the last service establishment state in each round: based on the actual total reward G and a preset threshold G... threshold And the preset bonus G add Update the actual total return G. Wherein, if the actual total return is greater than or equal to the threshold, then the actual total return = actual total return + the additional return; if the actual total return is less than the threshold, then the actual total return = actual total return - the additional return. That is, when G ≥ G... threshold At that time, G = G + G add The agent receives a positive additional reward, i.e., an extra bonus; when G < G threshold When, G = GG add The agent receives a negative additional reward, that is, it receives an additional penalty.
[0110] In some embodiments, Where G is the total reward obtained by the agent at the end of a round, and n is the total number of services to be built in the OTN network.
[0111] In some embodiments, such as Figure 4As shown, determining the pending services in the current service establishment state according to the action strategy includes the following steps:
[0112] Step 31: Calculate the probability of selecting each pending business under the current business establishment status.
[0113] In some embodiments, the probability of selecting each pending service in the current service establishment state is calculated according to the following formula:
[0114] Where p(s) tsvci Establish state S for the business t Select the i-th service Svc i The probability of establishing state S in the business is t. t The number of services already created is t = (0, ..., n-1), where n is the total number of services to be created in the OTN network. When t = n, the number of services to be created is 0, and the round ends.
[0115] Step 32: Determine a business to be built based on the probability of selecting each business to be built under the current business establishment status.
[0116] It should be noted that, based on the exploration approach of reinforcement learning, the selection of business to be built follows the randomness of the strategy.
[0117] Step 33: Sort the candidate routes of the identified services to be built according to the preset OTN network optimization objective function.
[0118] The objective function for OTN network optimization includes minimizing the routing cost of OTN network services or minimizing the latency of OTN network services.
[0119] The selection of alternative routes for the chosen service follows a greedy strategy, meaning all alternative routes are sorted according to the OTN network optimization objective function. For example, if the OTN network optimization objective function is to minimize routing cost, then based on the strategy of minimizing routing cost, all the identified alternative routes for the service can be sorted from smallest to largest according to their respective routing costs using KSP (K-Optimal Path Algorithm), RWA (Routing Wavelength Allocation Algorithm), and RSA (Asymmetric Encryption Algorithm).
[0120] Step 34: Calculate the selection probability of each candidate route in the sorting based on the number of candidate routes in the sorting.
[0121] In some embodiments, if the number of candidate routes in the sorting is m = 1, then the selection probability of that candidate route is: If the number of candidate routes in the sorting is m = 2, then the probability of selecting one of the candidate routes is: The probability of selecting the other alternative route is: If the number of candidate routes m in the sorting is greater than 2, then the selection probability of candidate routes 1 to m-1 is: The probability of selecting the m-th alternative route is: Where ε is the greed coefficient, 0.5 < ε < 1, and t is the business establishment state S. t The number of services already created is t = (0, ..., n-1), where n is the total number of services to be created in the OTN network, and i = (1, ..., m-1).
[0122] Step 35: Determine a candidate route based on the selection probability of each candidate route in the sorting, and use it as the route for the service to be established in the current service establishment state.
[0123] In some embodiments, the optimization target policy parameter can be the state behavior value Q. π (s,a), This indicates that the intelligent agent establishes state S from the business context. t Starting from the action strategy π, the expected cumulative reward obtained after performing action a is the cumulative reward.
[0124] In some embodiments, the optimization target policy parameter can also be the state value V. π (s), Q represents all state behavior values under the service establishment state S. π The weighted sum of (s,a). Where π(a|s) is the probability of executing action a according to action strategy π(s,a) under the service establishment state S, and A is the set of actions executed under each service establishment state.
[0125] In some embodiments, when the optimization target policy parameter is the state behavior value Q π When (s,a), the Monte Carlo Process (MCP) algorithm, the time-difference algorithm for different strategies (TD-Error for different strategies), or the time-difference algorithm for the same strategy (TD-Error for the same strategy) can be used to calculate and update the optimized target strategy parameters for each service establishment state. In some embodiments, the Q-Learning algorithm in the TD-Error algorithm for different strategies can be selected, or the SASA (State-Action-Reward-Action) algorithm in the TD-Error algorithm for the same strategy can be selected. Accordingly, updating the action strategy according to the optimal optimized target strategy parameters for each service establishment state (i.e., step 14) includes: according to the state behavior value Q π (s,a) Update the action policy.
[0126] For example, if the Q-Learning algorithm or the SASA algorithm is used, determining the optimal target strategy parameters for each service establishment state (i.e., step 13) may include: the optimal target strategy parameters (i.e., state behavior values Q) for each service establishment state in the preset number of rounds. π In (s,a)), the maximum value of the optimal target strategy parameter is determined for each service establishment state.
[0127] In some embodiments, when the optimization target policy parameter is the state value V π When (s), a dynamic programming algorithm can be used to calculate and update the optimization target strategy parameters. Correspondingly, updating the action strategy based on the optimal optimization target strategy parameters under each service establishment state (i.e., step 14) includes: based on the state value V... π (s) Update the action policy μ(s,a).
[0128] The following sections explain the process of optimizing OTN network resources using the Monte Carlo algorithm, Q-Learning algorithm, SASA algorithm, and dynamic programming algorithm, respectively.
[0129] (1) The process of optimizing OTN network resources using the exploratory initialization Monte Carlo algorithm is as follows:
[0130] Initialize the entire network topology environment, for all s∈S, a∈A(s),
[0131] Q(s,a)←0; The initial value of the action policy is μ(s,a);
[0132] returns(s,a) ←emptylist;
[0133] The repeat loop will repeat the following process:
[0134] {
[0135] Choose s0∈S, a0∈A(s) according to μ(s,a) and generate a new Episode;
[0136] For each pair (s, a) in this episode:
[0137] G←(s,a) represents the reward after the first occurrence;
[0138] Add G to the returns(s,a);
[0139] Let the state behavior value Q(s,a) ← average(returns(s,a)) be the average of the returns;
[0140] For each s in this episode:
[0141] π(s) ← argmax a Q(s,a);
[0142] }
[0143] (2) The process of optimizing OTN network resources using the Q-Learning (i.e., heterogeneous TD-Error) algorithm is as follows:
[0144] Initialize the entire network topology environment, for all s∈S, a∈A(s),
[0145] Q(s,a)←0; the action policy is μ(s,a);
[0146] Repeat the following process for each Episode loop:
[0147] Initialize the state space S;
[0148] Repeat (repeat the following process for each step in the loop of this episode):
[0149] According to strategy μ(s,a), in s t State selection action a t ;
[0150] Perform action a t and receive timely rewards R t+1 and the next state s t+1 ;
[0151] Let Q(s) t ,a t )←Q(s t ,a t )+α[R t+1 +γmax a Q(s t+1 ,a)-Q(s t ,a t )];
[0152] Where α is the learning rate;
[0153] s t ←s t+1 ;
[0154] Until s t The state is terminated;
[0155] Until all Q(s,a) converge;
[0156] Output the final strategy: π(s) ← argmax a Q(s,a);
[0157] (3) The process of optimizing OTN network resources using the SARSA (i.e., TD-Error with the same strategy) algorithm is as follows:
[0158] Initialize the entire network topology environment. For all s∈S, a∈A(s), Q(s,a)←0;
[0159] Repeat the following process for each Episode loop:
[0160] Initialize the state space S;
[0161] Given an initial state s0, and according to a greedy policy ε (choose the action that maximizes the immediate reward), select action a0;
[0162] Repeat (repeat the following process for each step in the loop of this episode):
[0163] According to the greedy strategy ε, in s t State selection action a t Receive timely rewards R t+1 and the next state s t+1 ;
[0164] Action a is obtained according to the greedy strategy ε. t+1 ;
[0165] Let Q(s) t ,a t )←Q(s t ,a t )+α[R t+1 +γQ(s t+1 ,a t+1 )-Q(s t ,a t )];
[0166] Where α is the learning rate;
[0167] s t ←s t+1 ;a t ←a t+1 ;
[0168] Until s t The state is terminated;
[0169] Until all Q(s,a) converge;
[0170] Output the final strategy: π(s) ← argmax a Q(s,a);
[0171] (4) The process of optimizing OTN network resources using a dynamic programming algorithm based on policy iteration is as follows:
[0172] Step 1: Initialize the entire network topology environment.
[0173] For all s t ∈S, a∈Α(s), V(s) t =0, let all The action policy is initialized to μ(s);
[0174] Step 2, Strategy Evaluation
[0175] Here p(s) t+1 ,R t+1 |s t μ(s) and p(s) t+1 ,R t+1 |s t a) indicates that strategy μ(s) is adopted in state s. t The probability of performing the corresponding action 'a';
[0176] The Repeat loop repeats the following process:
[0177] Δ←0;
[0178] For each s t ∈S:
[0179] v←V(s t );
[0180]
[0181] Δ←max(Δ,|vV(s t )|);
[0182] Convergence continues until Δ < θ (where θ is a specified constant);
[0183] Step 3, Strategy Improvement
[0184] For each s t ∈S:
[0185] a←μ(s);
[0186]
[0187] If a≠μ(s), then the policy is not convergent; otherwise, the policy is convergent.
[0188] If the policy converges, the algorithm terminates and returns V(s) and μ(s); otherwise, it continues to the processing in step 2.
[0189] Based on the same technical concept, this disclosure also provides an OTN network resource optimization device, such as... Figure 5 As shown, the OTN network resource optimization device includes: a first processing module 101, a second processing module 102, and an update module 103. The first processing module 101 is used to: determine the services to be established in the current service establishment state according to the action policy; create the services to be established; calculate the timely reward in the current service establishment state; enter the next service establishment state; until the end of one round; calculate and update the optimization target policy parameters in each service establishment state according to the timely reward in each service establishment state; and iterate for a preset number of rounds to calculate and update the optimization target policy parameters in each service establishment state.
[0190] The second processing module 102 is used to determine the optimal optimization target strategy parameters for each service establishment state based on the optimization target strategy parameters for each service establishment state in the preset number of rounds.
[0191] The update module 103 is used to update the action strategy according to the optimal optimization target strategy parameters of each service establishment state.
[0192] In some embodiments, the first processing module 101 is configured to calculate the expected return in the current business establishment state based on the timely rewards in each business establishment state after the next business establishment state; and calculate and update the optimization target strategy parameters in the current business establishment state based on the expected return in the current business establishment state.
[0193] In some embodiments, the first processing module 101 is configured to calculate the expected return under the current service establishment state according to the following formula: Among them, G t Establish state S for business t Next, execute action a t The expected return, γ is the discount factor, 0 < γ < 1; R is the immediate reward, and t is the business establishment state S. t The number of services already created, t = (0, ..., n-1), where n is the total number of services to be created in the OTN network.
[0194] In some embodiments, the expected return in the last business establishment state in each round is the actual total return of the round. The first processing module 101 is further configured to update the actual total return based on the actual total return, a preset threshold, and a preset additional return after calculating the actual total return in the last business establishment state in each round and before calculating the optimization target strategy parameters in the last business establishment state in each round; wherein, if the actual total return is greater than or equal to the threshold, then the actual total return = actual total return + the additional return; if the actual total return is less than the threshold, then the actual total return = actual total return - the additional return.
[0195] In some embodiments, the optimization target policy parameter includes the state behavior value Q. π (s,a), Alternatively, the optimization objective strategy parameters include the state value V. π (s), Where π(a|s) is the probability of taking action a according to action strategy π(s,a) under the service establishment state S, and A is the set of actions executed under each service establishment state.
[0196] In some embodiments, when the optimization target policy parameter is the state behavior value Q π When (s,a), the Monte Carlo algorithm, the time difference algorithm with different strategies, or the time difference algorithm with the same strategy are used to calculate and update the optimization target strategy parameters in each service establishment state.
[0197] The update module 103 is used to update the state behavior value Q. π (s,a) Update the action policy.
[0198] In some embodiments, when the optimization target policy parameter is the state value V π When (s), the optimization objective strategy parameters are calculated using a dynamic programming algorithm.
[0199] The update module 103 is used to update the state value V. π (s) Update the action policy.
[0200] In some embodiments, the first processing module 101 is configured to: calculate the probability of selecting each pending service in the current service establishment state; determine a pending service based on the probability of selecting each pending service in the current service establishment state; sort the candidate routes of the determined pending service according to a preset OTN network optimization objective function; calculate the selection probability of each candidate route in the sorted list based on the number of candidate routes in the sorted list; and determine a candidate route based on the selection probability of each candidate route in the sorted list as the route of the pending service in the current service establishment state.
[0201] In some embodiments, the first processing module 101 is used to calculate the probability of selecting each pending service in the current service establishment state according to the following formula: Where p(s) tsvci Establish state S for the business t Select the i-th service Svc i The probability of establishing state S in the business is t. t The number of services already created, t = (0, ..., n-1), where n is the total number of services to be created in the OTN network.
[0202] In some embodiments, the objective function for OTN network optimization includes minimizing the routing cost of OTN network services or minimizing the latency of OTN network services.
[0203] In some embodiments, the first processing module 101 is configured to, if the number of candidate routes in the sorting is m = 1, then the selection probability of the candidate route is: If the number of candidate routes in the sorting is m = 2, then the probability of selecting one of the candidate routes is: The probability of selecting the other alternative route is: If the number of candidate routes m in the sorting is greater than 2, then the selection probability of candidate routes 1 to m-1 is: The probability of selecting the m-th alternative route is: Where ε is the greed coefficient, 0.5 < ε < 1, and t is the business establishment state S. t The number of services already created is t = (0, ..., n-1), where n is the total number of services to be created in the OTN network, and i = (1, ..., m-1).
[0204] This disclosure also provides a computer device, which includes one or more processors and a storage device; wherein the storage device stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the OTN network resource optimization method provided in the foregoing embodiments.
[0205] This disclosure also provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed, implements the OTN network resource optimization method as provided in the foregoing embodiments.
[0206] It will be understood by those skilled in the art that all or some of the steps in the methods disclosed above, and the functional modules / units in the apparatus, can be implemented as software, firmware, hardware, and suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0207] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.
Claims
1. An OTN network resource optimization method, characterized in that, The method comprises the following steps: determining a to-be-established service in a current service establishment state according to an action strategy, creating the to-be-established service, and calculating an immediate reward in the current service establishment state, entering a next service establishment state, until a round is finished, calculating and updating an optimal target strategy parameter in each service establishment state according to the immediate reward in each service establishment state; iterating a preset number of rounds to calculate and update the optimal target strategy parameter in each service establishment state; determining an optimal optimal target strategy parameter in each service establishment state according to the optimal target strategy parameter in each service establishment state in the preset number of rounds; updating the action strategy according to the optimal optimal target strategy parameter in each service establishment state; the step of calculating and updating the optimal target strategy parameter in each service establishment state according to the immediate reward in each service establishment state comprises the steps of: calculating an expected return in the current service establishment state according to the immediate reward in each service establishment state after the next service establishment state; calculating and updating the optimal target strategy parameter in the current service establishment state according to the expected return in the current service establishment state; the expected return in the last service establishment state in each round is an actual total return of the round, and the method further comprises the steps of: updating the actual total return according to the actual total return, a preset threshold and a preset additional return; wherein if the actual total return is greater than or equal to the threshold, the actual total return = the actual total return + the additional return; if the actual total return is less than the threshold, the actual total return = the actual total return - the additional return.
2. The method of claim 1, wherein, the expected return in the current service establishment state is calculated according to the following formula: wherein G t is the state of service establishment S t , the expected return of performing action a t , γ k is the discount factor of R t+k+1 , 0 < γ k < 1; R t+k+1 is the immediate reward obtained in the state of service establishment s t+k+1 , t is the number of services created in the state of service establishment S t , t = (0,..., n-1), and n is the total number of services to be established in the OTN network.
3. The method of claim 1, wherein, The optimization target policy parameter includes a state-action value Q π (s, a), Alternatively, The optimization target policy parameter comprises a state value V π (s), Wherein, π(a|s) is the probability of taking action a according to the action policy π(s,a) in the service establishment state S, and A is the set of actions performed in each service establishment state; wherein, is the expected cumulative return obtained after performing action a according to the action policy π(s,a) from the state S t , R t+k+1 is the immediate reward obtained in the state s t+k+1 , γ k is a discount coefficient of R t+k+1 .
4. The method of claim 3, wherein, When the optimization target policy parameter is a state-action value Q π (s,a), the optimization target policy parameter in each service establishment state is calculated and updated using a Monte Carlo algorithm, an off-policy time-difference algorithm, or an on-policy time-difference algorithm. The action policy is updated according to the optimal optimization target strategy parameter in each service establishment state, including: updating the action policy according to the state behavior value Q π (s,a) 5. The method of claim 3, wherein, when the optimization target policy parameter is a state value V π (s), the dynamic programming algorithm is used to calculate the optimization target policy parameter; The action policy is updated according to the optimal optimization target strategy parameter in each service establishment state, including: updating the action policy according to the state value V π (s) updating the action policy.
6. The method according to any one of claims 1 to 5, wherein, the step of determining the to-be-established service in the current service establishment state according to the action strategy comprises the steps of: calculating a probability of selecting each to-be-established service in the current service establishment state; determining a to-be-established service according to the probability of selecting each to-be-established service in the current service establishment state; sorting candidate routes of the determined to-be-established service according to a preset OTN network optimization target function; calculating a selection probability of each candidate route in the sorting according to the number of candidate routes in the sorting; determining a candidate route as a route of the to-be-established service in the current service establishment state according to the selection probability of each candidate route in the sorting.
7. The method of claim 6, wherein, The probability of selecting each service to be established under the current service establishment state is calculated according to the following formula: Wherein, p(s tsvci ) is the probability of selecting the ith service Svc i under the service establishment state S t , t is the number of services established under the service establishment state S t , t=(0,…,n-1), and n is the total number of services to be established in the OTN network.
8. The method of claim 6, wherein, The OTN network optimization target function comprises a minimum OTN network service routing cost or a shortest OTN network service delay.
9. The method of claim 6, wherein, the step of calculating the selection probability of each candidate route in the sorting according to the number of candidate routes in the sorting comprises the steps of: If the number of the candidate routes m in the ranking is 1, the selection probability of the candidate route is: wherein μ(S t , a t1 ) is the selection probability of 1 candidate route of the (t+1)th service when performing action a t under service establishment state S t1 , to establish the (t+1)th service. If the number of candidate routes m in the ranking is 2, the selection probability of one of the candidate routes is: The selection probability of the other candidate route is: wherein μ(S t ,a t1 ) is the selection probability of the first candidate route of the two candidate routes of the (t+1)th service when the (t+1)th service is established under the service establishment state S t and the action a t1 ; μ(S t ,a t2 ) is the selection probability of the second candidate route of the two candidate routes of the (t+1)th service when the (t+1)th service is established under the service establishment state S t and the action a t2 . If the number of the candidate routes m in the ranking is greater than 2, the selection probability of the first to the m-1th candidate routes is: The selection probability of the mth candidate route is: Wherein, μ(S t ,a tm ) is the selection probability of the mth candidate route when action a tm is performed under the service establishment state S t , and the (t+1)th service is established. Wherein, ε is the greed coefficient, 0.5<ε<1, t is the service establishment state S t The number of services created below, t=(0,...,n-1), n is the total number of services to be built in the OTN network, i=(1,...,m-1).
10. An OTN network resource optimization apparatus, comprising: a first processing module, a second processing module and an updating module, The first processing module is configured to determine a to-be-established service in a current service establishment state according to the action policy, create the to-be-established service, and calculate an immediate reward in the current service establishment state, enter a next service establishment state, until a round ends, and calculate and update an optimal target policy parameter in each service establishment state according to the immediate reward in each service establishment state; iterate a preset number of rounds to calculate and update the optimal target policy parameter in each service establishment state; wherein an expected return in the current service establishment state is calculated according to the immediate reward in each service establishment state after the next service establishment state, and the optimal target policy parameter in the current service establishment state is calculated and updated according to the expected return in the current service establishment state; an expected return in a last service establishment state in each round is an actual total return of the round, after the actual total return in the last service establishment state in each round is calculated, before the optimal target policy parameter in the last service establishment state in each round is calculated, the actual total return is updated according to the actual total return, a preset threshold, and a preset additional return; wherein if the actual total return is greater than or equal to the threshold, the actual total return = the actual total return + the additional return; if the actual total return is less than the threshold, the actual total return = the actual total return - the additional return; The second processing module is configured to determine an optimal optimal target policy parameter in each service establishment state according to the optimal target policy parameter in each service establishment state in the preset number of rounds. The updating module is configured to update the action policy according to the optimal optimal target policy parameter in each service establishment state.
11. A computer device, comprising: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the OTN network resource optimization method according to any one of claims 1-9.
12. A computer readable medium having stored thereon a computer program, wherein, The program is executed to implement the OTN network resource optimization method according to any one of claims 1-9.