Network energy efficiency optimization method, computer equipment and computer readable medium

Abstract

The invention provides a network energy efficiency optimization method. A digital twin reinforcement learning network model is used for learning to obtain action strategies of multiple to-be-optimized services; traversing all service establishment states according to an action strategy, determining a service path establishment action with the optimal global energy efficiency in each service establishment state, and obtaining a service path establishment action sequence with the optimal global energy efficiency; establishing an action sequence according to the service path with the optimal global energy efficiency, and determining the path distribution with the optimal energy efficiency of each to-be-optimized service; according to the embodiment of the invention, the problem of low energy efficiency in network operation is effectively identified by adopting a digital twin modeling and analysis technology, and the path distribution with the optimal global energy efficiency of each service is determined, so that the path with the optimal energy efficiency is configured for each service, and the energy efficiency use is optimized and improved. The invention further provides computer equipment and a computer readable medium.

Description

Technical Field

[0001] The present disclosure relates to the field of self-intelligent optical network technology, and in particular to a network energy efficiency optimization method, a computer device, and a computer-readable medium. Background Art

[0002] The global communications industry is moving from the Internet era and the cloud era to the intelligent era. Faced with market competition, industry-wide digital transformation, and increasing network complexity, challenges such as network planning, optimization, operation and maintenance, reliability, agility, and OPEX (Operating Expense) reduction have made it inevitable for optical networks to develop in an intelligent direction. The emergence and rise of intelligent optical network technology has provided a solid foundation and the main means for achieving the goal of intelligent optical networks.

[0003] Autonomous optical network technology originates from the concept of "Autonomous Networks" (AN) proposed by TM Forum in 2019. After two years, the AN concept has reached a consensus in the industry. It aims to enable the digital transformation of operators' networks through the integration of network technology and digital technology, and provide innovative ICT services and user experience with zero waiting, zero contact, and zero failure for vertical industries and consumer users, and create self-configuration, self-repair, and self-optimization network capabilities for the entire life cycle of operators' network operations. Therefore, the autonomous optical network technology derived and developed on this basis can achieve the intelligent management, ultimate experience, and flexible and open intelligent goals of optical networks through self-configuration, self-recovery, and self-optimization intelligent operation and maintenance capabilities to meet customer needs for improving user experience and reducing operating costs.

[0004] Among the many intelligent capabilities that an intelligent optical network (AN OTN) needs to have, energy efficiency problem identification and optimization is the key core capability to achieve OTN network intelligence. As the trend of global warming continues to worsen, energy conservation and consumption reduction have great international economic and political significance.

[0005] At the same time, the emerging digital twin (DT) technology has also entered a period of rapid development and has been widely used in manufacturing, supply chain and other fields to support industrial digital transformation. The digital transformation in the telecommunications field also has a strong demand for digital twin technology, which is considered by the industry to be the foundation of digital transformation and an important support and component for realizing AN self-intelligent network architecture and technology. The network digital twin accurately perceives the state of itself and the external environment, establishes a digital mirror of the internal and external environment, integrates simulation and preventive prediction capabilities, and plays a key enabling technology role in scenarios such as "low-cost trial and error", "intelligent decision-making", and "predictive maintenance".

[0006] Digital twin, or DT technology, can be understood as building a digital (virtual) model of a physical object in a digital space, and using data from the physical object to continuously correct the model and update the model status, so that it is consistent with the physical object throughout its life cycle, and can mirror the real-time operating status of the physical object with high fidelity. The model becomes the digital twin of the physical object (abbreviated as digital twin). Based on digital twins, monitoring, analysis, prediction, diagnosis, training, and simulation can be performed, and the simulation results can be fed back to the physical object to help optimize and make decisions on the physical object. Related technologies involving digital twin model construction, real-time update of digital model status, simulation analysis and control decision-making based on digital twins can be collectively referred to as digital twin DT technology. From this, it can be seen that how to combine the structural characteristics and state characteristics of the physical object mirrored by the twin and the application scenarios of DT to build a DT model is a key technology of DT.

[0007] In summary, how to use digital twin DT modeling and analysis technology to effectively identify low energy efficiency problems in OTN network operations and optimize and improve energy efficiency, and try to minimize the impact of energy efficiency optimization methods and measures on existing network operations and make users unaware, has become the current industry's focus and self-intelligence issue to be solved. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the present disclosure provides a network energy efficiency optimization method, a computer device and a computer-readable medium.

[0009] In a first aspect, an embodiment of the present disclosure provides a network energy efficiency optimization method, including:

[0010] Determine the action strategies of multiple services to be optimized according to a pre-trained action strategy learning model; wherein the action strategy learning model is a digital twin deep reinforcement learning network model, and the energy consumption optimization objective function of the action strategy learning model is to minimize the sum of the workload energy consumption and the equipment energy consumption of the multiple services to be optimized; the actual total reward of a trajectory of the action strategy learning model is the timely reward in the establishment state of the last service in the trajectory;

[0011] According to the action strategy, traverse all service establishment states, determine the service path establishment action with the best global energy efficiency in each service establishment state, and obtain the service path establishment action sequence with the best global energy efficiency;

[0012] An action sequence is established according to the service path with the best global energy efficiency, and the path distribution with the best global energy efficiency of the multiple services to be optimized is determined.

[0013] In another aspect, the present disclosure also provides a computer device, including:

[0014] one or more processors;

[0015] a storage device having one or more programs stored thereon;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the network energy efficiency optimization method as described above.

[0017] On the other hand, an embodiment of the present disclosure further provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed, the network energy efficiency optimization method as described above is implemented.

[0018] The network energy efficiency optimization method and device provided by the embodiments of the present disclosure utilize a digital twin reinforcement learning network model to learn action strategies for multiple services to be optimized, wherein the energy consumption optimization objective function of the action strategy learning model is to minimize the sum of the workload energy consumption and the equipment energy consumption of each service to be optimized, and the action strategy learning model adopts a delayed reward mechanism; according to the action strategy, all service establishment states are traversed, and the service path establishment action with the best global energy efficiency under each service establishment state is determined to obtain the service path establishment action sequence with the best global energy efficiency; according to the service path establishment action sequence with the best global energy efficiency, the path distribution with the best global energy efficiency for each service to be optimized is determined; the embodiments of the present disclosure utilize digital twin modeling and analysis technology to effectively identify low energy efficiency problems occurring in network operations, and determine the path distribution with the best global energy efficiency for each service, so as to configure the path with the best global energy efficiency for each service and optimize and improve energy efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the network energy efficiency optimization process provided by the embodiment of the present disclosure Figure 1 ;

[0020] Figure 2 A schematic diagram of an action strategy learning model trained using a Q-Based Actor-Critic algorithm provided in an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of a flow chart of an action for establishing a service path with the best global energy efficiency under a service establishment state provided by an embodiment of the present disclosure;

[0022] Figure 4a-4e A schematic diagram of an operation of establishing a service path with the best global energy efficiency using the MCTS algorithm provided in an embodiment of the present disclosure;

[0023] Figure 5 Schematic diagram of the network energy efficiency optimization process provided by the embodiment of the present disclosure Figure 2 ;

[0024] Figure 6A schematic diagram of a resource interlocking optimization scenario provided by an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of a resource independent optimization scenario provided by an embodiment of the present disclosure;

[0026] Figure 8 A schematic diagram of a process for optimizing a service path to be optimized in a resource interlocking optimization scenario provided in an embodiment of the present disclosure;

[0027] Fig. 9 A schematic diagram of optimizing a service path to be optimized in a resource interlocking optimization scenario provided by a specific example of the present disclosure;

[0028] Fig.10 for Fig. 9 Schematic diagram of unlocking time and optimization time of each service path to be optimized;

[0029] Fig.11 A schematic diagram of a process for optimizing a service path to be optimized in a resource independent optimization scenario provided by an embodiment of the present disclosure;

[0030] Fig.12 A schematic diagram of optimizing a service path to be optimized in a resource independent optimization scenario provided by a specific example of the present disclosure;

[0031] Fig.13 for Fig.12 Schematic diagram of unlocking time and optimization time of each service path to be optimized;

[0032] Fig.14 A schematic diagram of a network energy efficiency optimization process provided for a specific example of the present disclosure. DETAILED DESCRIPTION

[0033] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0034] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0035] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.

[0036] The embodiments described herein may be described with reference to plan views and / or cross-sectional views by means of idealized schematic diagrams of the present disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings have schematic properties, and the shapes of the regions shown in the figures illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.

[0037] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0038] In view of the core capability of self-intelligent networks, namely, identification and optimization of energy efficiency issues in OTN networks, the embodiments of the present disclosure provide a method mechanism for identifying and optimizing energy efficiency issues in OTN networks using digital twin modeling technology. The method mechanism can be applied to networks such as OTN, PTN (Packet Transport Network), POTN (Packet Optical Transport Network), SPN (Slicing Packet Network), and IP. The embodiments of the present disclosure take OTN networks as an example for illustration.

[0039] Assuming that n OTN services are created under a given OTN network diagram topology (including mesh, star and other structural types), in the embodiment of the present disclosure, creating an OTN service refers to creating an ODU (Optical Channel Data Unit) layer service of the OTN, and the OCH (Optical Channel with full functionality) channel resources of the network topology fully meet the creation needs of creating these n OTN services, where n>1. The action strategy learning model for creating the n OTN services is defined as follows:

[0040] 1. Define the OTN network optimization objective function

[0041] The objective function of OTN network optimization can be to minimize the sum of the workload energy consumption and equipment energy consumption of the OTN service to be optimized. MinE , that is, the minimum energy consumption required by the entire network to support and meet the normal operation of the n OTN services within a specified time period, in kilowatt-hours. In the embodiment of the present disclosure, the OTN network optimization objective function is an energy consumption optimization objective function, which is expressed as follows:

[0042]

[0043] Among them, E(Svc i ) indicates that in the current trajectory of creating n services, the i-th OTN service Svc i The workload energy consumption, i.e., the node / port energy consumption used by the workload, is related to the energy efficiency ratio of the node / port through which the i-th OTN service path passes;

[0044] Represents the Topo of the entire network A The energy consumption of the equipment required to support the normal operation of the n OTN services. For example, if a network node only carries one OTN service, and this service only occupies part of the time slot resources on the line-side OCH channel, but in order to ensure the normal operation of this service, the relevant equipment of the network node (main control board, etc.) needs to be powered on, and the OTU board and port corresponding to the OCH channel also need to be powered on. This consumes energy in addition to the energy required by the workload of the OTN service. Therefore It can also be called extra energy consumption. A The total additional energy consumption of all n OTN services is

[0045] 2. Define the state

[0046] Status S t(t=0,…,n-1) indicates that t OTN services have been created (including detailed information of all services that have been created), (nt) OTN services have not been created (including detailed information of all uncreated OTN services), and the (t+1)th OTN service is about to be created.

[0047] Status S n Indicates that the creation of n OTN services for the entire network is completed.

[0048] 3. Define the trajectory

[0049] Using a certain action strategy to complete the sequential creation of n OTN services in the entire network is defined as a trajectory.

[0050] 4. Define immediate rewards and final return mechanisms

[0051] R t (t=0,…,n-1) represents the state S t Taking "minimum energy consumption of the entire network service" as an example of the OTN network optimization objective function, the embodiment of the present disclosure adopts a delayed reward mechanism here, R t The value is 0.

[0052] G represents the total reward obtained by the agent at the end of a trajectory: R n = G. The value of G can be defined as the reciprocal of the energy consumption value of the entire network service (here the energy consumption unit, kilowatt-hour, is removed) multiplied by 1000, that is, n represents the total number of OTN services in the entire network, so the service distribution solution with lower energy consumption has higher total return.

[0053] G t Indicates that in state S t Next, perform action a t The expected return, Where γ is the discount coefficient, 0<γ<1.

[0054] 5. Define actions and action strategies

[0055] An action t (t=0,…,n-1) means that, in the current network topology environment, among the services to be built, one service to be built is selected as the next service to be created, and one is selected from multiple alternative routes of the service to be built (routes to which network resources have been allocated, including latency, bandwidth, availability, energy consumption and other resources) as the resource route of the service and the process of creating the service is completed.

[0056] Multiple candidate resource routes for a certain OTN service to be built can be sorted according to the optimization strategy corresponding to the OTN network optimization objective function. If the OTN network optimization objective function is to minimize the energy consumption of the entire network, then the multiple candidate resource routes for the OTN service to be built can be sorted in order from small to large according to the energy consumption.

[0057] SvcID t+1 Indicates that in the current track of creating n services, in state S t Next (t+1)th OTN service Svc to be built t+1 The business ID of the action a t Decide.

[0058] The action strategy π(s,a) (also written as π(s)) represents the probability of selecting the OTN service to be built and selecting a certain candidate resource route under the condition that the OTN service to be built is selected. It represents the creation order of the OTN service to be built (including the route of the service to be built).

[0059] 6. Define state behavior value function and state value function

[0060] The state behavior value function is expressed as: It represents the expected reward obtained by the agent after starting from state S and performing action a according to the action strategy π(s,a).

[0061] The state value function is expressed as: It represents all state behavior values ​​Q under state S π The weighted sum of (s,a). Among them, π(a|s) represents the probability of executing action a according to the action strategy π(s,a) in state S.

[0062] Figure 1 Schematic diagram of the network energy efficiency optimization process provided by the embodiment of the present disclosure Figure 1 ,like Figure 1 As shown, the network energy efficiency optimization method includes the following steps:

[0063] Step S11, determining the action strategies of multiple services to be optimized based on the pre-trained action strategy learning model; wherein the action strategy learning model is a digital twin deep reinforcement learning network model, and the energy consumption optimization objective function of the action strategy learning model is to minimize the sum of the workload energy consumption and the equipment energy consumption of the multiple services to be optimized; the actual total return of a trajectory of the action strategy learning model is the timely reward for the last service establishment state in the trajectory.

[0064] In the disclosed embodiment, the action strategy learning model is a deep reinforcement learning network model (DRL) generated by digital twin technology modeling, and the convergence function of the action strategy learning model is an energy consumption optimization objective function. The other parameter definitions of the action strategy learning model are as described above and will not be repeated here.

[0065] Step S12, according to the action strategy, traverse all service establishment states, determine the service path establishment action with the best global energy efficiency in each service establishment state, and obtain the service path establishment action sequence with the best global energy efficiency.

[0066] In the whole network Topo A In the topological environment, the action strategy learned in step S11 can be further fine-tuned using the Monte Carlo Tree Search (MCTS) algorithm to obtain the globally energy-efficient service path establishment action under each service establishment state. Based on the globally energy-efficient service path establishment action under each service establishment state, a complete globally energy-efficient action sequence can be obtained.

[0067] Step S13, establishing an action sequence according to the service path with the best global energy efficiency, and determining the distribution of paths with the best global energy efficiency for multiple services to be optimized.

[0068] By establishing an action sequence based on the service path with the best global energy efficiency, the optimal energy efficiency distribution of n services to be optimized can be obtained.

[0069] The network energy efficiency optimization method and device provided by the embodiments of the present disclosure utilize a digital twin reinforcement learning network model to learn action strategies for multiple services to be optimized, wherein the energy consumption optimization objective function of the action strategy learning model is to minimize the sum of the workload energy consumption and the equipment energy consumption of each service to be optimized, and the action strategy learning model adopts a delayed reward mechanism; according to the action strategy, all service establishment states are traversed, and the service path establishment action with the best global energy efficiency under each service establishment state is determined to obtain the service path establishment action sequence with the best global energy efficiency; according to the service path establishment action sequence with the best global energy efficiency, the path distribution with the best global energy efficiency for each service to be optimized is determined; the embodiments of the present disclosure utilize digital twin modeling and analysis technology to effectively identify low energy efficiency problems occurring in network operations, and determine the path distribution with the best global energy efficiency for each service, so as to configure the path with the best global energy efficiency for each service and optimize and improve energy efficiency.

[0070] In the whole network Topo A In the topology environment, the Actor-Critic algorithm can be used to obtain the Q for optimizing the energy efficiency distribution of n services. ω (s,a) state-value neural network and π θ(s,a) Strategy neural network, using Actor-Critic algorithm modeling and model processing flow is as follows:

[0071] According to the design in the Actor algorithm, the parameterized action strategy function is defined as π θ (s,a),π θ The value of (s,a) can be obtained from the trained π used in the Actor algorithm. θ The value is inferred by the deep neural network, where θ is the parameter vector of the neural network layer.

[0072] According to the design in the Critic algorithm, the parameterized state behavior value function is defined as Q ω (s,a)=φ(s,a) T ω, Q ω The value of (s,a) can be inferred from the trained Q-value deep neural network used for the Critic algorithm, where ω is the neural network layer parameter vector, and φ(s,a) is the feature vector function of state S and action a, which can be used as the input to the Critic deep neural network.

[0073] Figure 2 Schematic diagram of an action strategy learning model trained by the Q-Based Actor-Critic (QAC) algorithm provided in an embodiment of the present disclosure. The Q-Based Actor-Critic algorithm is one of the typical algorithms of DRL, such as Figure 2 As shown in FIG, the process of using the QAC algorithm combined with the action strategy learning model to determine the action strategy of the service to be optimized is as follows:

[0074] Initialize the entire network topology environment, including initializing s0∈S and action strategy parameter vector θ.

[0075] Step 1: Initialize sampling actions a~π according to the action strategy θ ;

[0076] Step 2: Calculate the score Q of action a ω (s,a),Q ω (s,a)=φ(s,a) T ω;

[0077] For each step of the sampling action, the following steps are performed:

[0078] Step 3: Sample and reward in time , sample the next state transition s'~P s a ;

[0079] Step 4: Sample the next action a'~π according to the action strategy θ(s',a');

[0080] Step 5: Calculate the score Q of action a' ω (s',a'),Q ω (s',a')=φ(s',a') T ω;

[0081] Step 6, calculate the intermediate value δ, δ = r + γQ ω (s',a')-Q ω (s,a);

[0082] Step 7: Determine the action strategy parameter vector θ. The action strategy parameter vector θ is the Actor model parameter;

[0083] Step 8: Determine ω←ω+βδφ(s,a) based on the intermediate value δ and the constant β, where ω is the Critic model parameter;

[0084] Step 9, update the action and state, a←a',s←s';

[0085] End for;

[0086] End ends the processing.

[0087] In some embodiments, a Monte Carlo tree search algorithm is used to traverse all service establishment states according to the action strategy to determine the service path establishment action with the best global energy efficiency in each service establishment state. Figure 3 As shown, for each service establishment state, a Monte Carlo tree search algorithm is used to traverse all service establishment states according to the action strategy to determine the service path establishment action with the best global energy efficiency under the service establishment state, including the following steps:

[0088] Step S21, calculate the action score corresponding to each alternative business path establishment action in the business establishment state according to the score of executing the alternative business path establishment action, the policy score of the action policy in the business establishment state, the number of times the alternative business path establishment action is selected in the business establishment state and the preset hyperparameters.

[0089] A service establishment state St may correspond to one or more alternative service path establishment actions. The next service establishment state S obtained by executing different alternative service path establishment actions under the service establishment state St is t+1 In order to select the service path with the best global energy efficiency, the most appropriate service path establishment action must be selected from each candidate service path establishment action in each service establishment state. In this step, each candidate service path establishment action is scored to obtain a corresponding action score.

[0090] Step S22, determine the target action corresponding to the maximum value of the action score, execute the target action, obtain the next business establishment state, and generate subsequent business establishment states from the next business establishment state in sequence according to the action strategy until the trajectory ends.

[0091] The alternative service path establishment action with the highest action score is selected from all the alternative service path establishment actions in the service establishment state St as the execution action for the next step of expansion, and the selected alternative service path establishment action is executed to obtain the next service establishment state S t+1 By analogy, each subsequent service establishment state is gradually generated until the service establishment state S is generated. n , until the current trajectory ends.

[0092] Step S23, calculating the state score of the next business establishment state according to the state value of the next business establishment state and the timely reward obtained from the next business establishment state to the end of the trajectory.

[0093] Step S24, based on executing the alternative service path establishment action in the service establishment state, obtain the value of the state score record of the next service establishment state and the state scores corresponding to the state score record in the next service establishment state, calculate the average state score of the next service establishment state, and use the average score to update the score of executing the alternative service path establishment action.

[0094] This step performs a backtracking operation and executes multiple actions a in the business establishment state St t Multiple evaluation simulations are performed to obtain corresponding multiple state scores, thereby generating a state score record, which includes multiple state scores. The value L of the state score record is the value of executing the alternative service path establishment action a under the service establishment state St. t In this step, the average state score is calculated based on the sum of the state scores and the value L of the state score record, and the average state score is used to update the execution of the alternative service path establishment action a. t score.

[0095] Step S25, after traversing all service establishment states for a preset number of times, determine the service path establishment action with the best global energy efficiency in the service establishment state according to the number of times each candidate service path establishment action is selected in the service establishment state.

[0096] After a preset number of Monte Carlo tree search processes, the alternative service path establishment actions a under the service establishment state St are obtained. t The number of times N(a t ), select N(a t ) The largest value of the alternative service path establishment action a t, as the alternative service path establishment action that the Agent needs to select in the service establishment state St. The preset number of times can be determined according to the algorithm application needs.

[0097] Figure 4a-4e The schematic diagram of the operation of establishing a service path with the best global energy efficiency using the MCTS algorithm provided in the embodiment of the present disclosure is as follows: Figure 4a-4e The process of using the MCTS algorithm to determine the business path establishment action with the best global energy efficiency is described in detail.

[0098] like Figure 4a As shown, there are three alternative service path establishment actions in the service establishment state St, namely a t =a1, a t =a2, a t =a3, and the action scores corresponding to the above three candidate service path establishment actions are calculated respectively according to the following formula (1):

[0099]

[0100] Among them, U score (a t ) is the action score corresponding to the action of establishing the alternative service path; Q(a t ) establishes action a for executing alternative business paths obtained through MCTS search t score; η is a hyperparameter, which is adjusted manually; π(a t |s t ) is the action strategy π(s t ) In the service establishment state St, the corresponding alternative service path establishment action a t Strategy score; N(a t ) Select an alternative service path to establish action a for the service establishment state St t The number of times.

[0101] According to the above formula (1), the action scores corresponding to the above three alternative service path establishment actions are calculated respectively, U score (a1) = 200, U score (a2) = 300, U score (a3)=250.

[0102] like Figure 4b As shown, the alternative service path establishment action a2 with the highest action score is selected as the next extended execution action, and the next service establishment state S is generated accordingly. t+1 .

[0103] like Figure 4c As shown, the action strategy π(s t ), from the next business establishment state St+1 Start the extended execution (FastRollout) until the business establishment status S is generated n The next service establishment state S is calculated according to the following formula (2): t+1 Status score:

[0104]

[0105] Among them, V score (s t+1 ) Establish state S for the next business t+1 Status score; V π (s t+1 ) is the action strategy π(s) t ), the next business establishment state S t+1 Status value of R n To adopt the action strategy π(s t ), from the next business establishment state S t+1 To the end of the track (ie, generating the business establishment state S n ), the immediate reward obtained, R n =G.

[0106] like Figure 4d As shown, perform a backtracking operation and perform an alternative service path establishment action a t , perform multiple evaluation simulations to establish state S from the next business t+1 Expand each V score(l) (S t+1 ) to perform cumulative averaging, calculate and update the alternative action a t Q(a t ) value. The next service establishment state S is calculated according to the following formula (3): t+1 Average status score:

[0107]

[0108] Among them, Q(a t ) Establish state S for the next business t+1 The average state score of L is the action a of executing the alternative service path establishment in the service establishment state St. t Get the next service establishment status S t+1 The value of the status score record.

[0109] like Figure 4e As shown in the figure, after a preset number of Monte Carlo tree search processes, the action selection decision of the Agent in the service establishment state St follows the following principles: a t = arg maxN(a t), where N(a t ) is the establishment action a of each alternative service path under the service establishment state St t The number of times selected. That is, select N(a t ) value is the alternative service path establishment action that the Agent needs to select in the service establishment state St.

[0110] Based on the above steps, the above MCTS algorithm can be used for each service establishment state to obtain the service path establishment action that needs to be selected in each service establishment state, that is, in the entire network Topo A In the topological environment, the service path establishment action with the best global energy efficiency under each service establishment state is determined, thereby obtaining the path distribution with the best global energy efficiency for n services to be optimized.

[0111] In some embodiments, Figure 5 As shown, after establishing an action sequence according to the service path with the best global energy efficiency and determining the path distribution with the best global energy efficiency of the multiple services to be optimized (i.e., step 13), the network energy efficiency optimization method may further include the following steps:

[0112] Step S14: Determine multiple service paths to be optimized according to the distribution of paths with the best global energy efficiency of the multiple services to be optimized.

[0113] Step S15, optimizing each service path to be optimized one by one.

[0114] In some embodiments, the optimizing each of the service paths to be optimized one by one (i.e., step S15) includes the following steps:

[0115] Step S151 , dividing the service path to be optimized into optimization scenarios, where the types of optimization scenarios include resource interlocking optimization scenarios and resource independent optimization scenarios.

[0116] The service paths to be optimized can be divided into multiple different types of optimization scenarios, that is, multiple service paths to be optimized can be divided into resource interlocking optimization scenarios and resource independent optimization scenarios, and the OTN service paths to be optimized in the optimization scenarios can be optimized for different optimization scenarios. It should be noted that multiple service paths to be optimized can also be divided into one type of optimization scenario, in which case, each service path to be optimized in the optimization scenario is optimized.

[0117] Figure 6 A schematic diagram of a resource interlocking optimization scenario provided by an embodiment of the present disclosure, such as Figure 6 As shown in the figure, there are three OTN service paths in the OTN network topology: Path1, Path2, and Path3. Figure 6As shown by the three solid lines in the figure, the DRL&MCTS algorithm determines the optimal path distribution for the global energy efficiency of these three OTN services as follows: Figure 6 As shown by the three dotted lines in the figure, there is a resource occupation conflict between the source path of Path2 and the target path Path1' after optimization of Path1 in the AJ segment link; there is a resource occupation conflict between the source path of Path3 and the target path Path2' after optimization of Path2 in the AL segment link; there is a resource occupation conflict between the source path of Path1 and the target path Path3' after optimization of Path3 in the AB segment link.

[0118] Figure 7 A schematic diagram of a resource independent optimization scenario provided by an embodiment of the present disclosure, such as Figure 7 As shown in the figure, there are three OTN service paths in the OTN network topology: Path1, Path2, and Path3. Figure 6 As shown by the three solid lines in the figure, the DRL&MCTS algorithm determines the optimal path distribution for the global energy efficiency of these three OTN services as follows: Figure 7 As shown by the three dotted lines in the figure, there will be no resource conflict in the progressive optimization process of the three OTN services from the source path to the target path. The resource optimization process of each OTN service is relatively independent and will not be restricted by the resource occupation of other OTN service paths.

[0119] Step S152: for each optimization scenario, optimizing each service path to be optimized in the optimization scenario one by one.

[0120] According to the type of each optimization scenario, an optimization strategy matching the type of the optimization scenario is selected to perform path migration, so as to migrate each service path to be optimized to a corresponding target path to achieve energy efficiency optimization.

[0121] It can be seen from the above steps S151-S152 that in the embodiment of the present disclosure, the DRL&MCTS algorithm technology is used to obtain W service energy efficiency optimization target paths in the specified network topology; according to the progressive optimization scheme, the W service energy efficiency optimization process is decomposed into two resource optimization scenarios, namely, the resource interlocking optimization scenario and the resource independent optimization scenario, wherein the number of resource interlocking optimization scenarios is m, m>=1, and the number of resource independent optimization scenarios is n, n>=1; progressive energy efficiency optimization processing is performed on the service paths in the m resource interlocking optimization scenarios and the n resource independent optimization scenarios, respectively. In this way, progressive energy efficiency optimization can be performed on batch service paths in the specified network topology.

[0122] Due to the problem of resource interlocking, it is impossible to optimize each service path to be optimized one by one. For this reason, the embodiment of the present disclosure proposes a method mechanism for gradually optimizing the energy efficiency of each service by rerouting one of the service paths to be optimized to a temporary path to release the resource interlocking.

[0123] In some embodiments, Figure 8 As shown, for the resource interlocking optimization scenario, optimizing each service path to be optimized in the optimization scenario one by one includes the following steps:

[0124] Step S31, determining an unlocking path and a rerouting path of the unlocking path according to the resource interlocking state.

[0125] The unlocking path is one of the business paths to be optimized in the resource interlocking optimization scenario. According to the resource interlocking state, a business path to be optimized (i.e., the source path) is found for unlocking the resource interlocking state, and a rerouting path for the business path to be optimized is determined. The starting end node and the ending end node of the rerouting path correspond to the starting end node and the ending end node of the unlocking path, respectively.

[0126] Step S32, determining the rerouting time of the unlocked path, and migrating the unlocked path to the rerouting path according to the rerouting time to release the network node.

[0127] In this step, the migration of the unlocked path is completed, and the network nodes occupied by the unlocked path and required to be occupied by the target paths of other service paths to be optimized are correspondingly released.

[0128] Step S33, based on the network nodes released by the previous path migration, determine the business path to be optimized for this path migration and the optimization time of the business path to be optimized, and migrate the business path to be optimized for this path migration to the corresponding target path according to the optimization time until all the remaining business paths to be optimized in the resource interlocking optimization scenario complete the path migration.

[0129] After the unlocked path migration is completed, the service path to be optimized and its optimization time that need to be migrated are determined according to the network node released in step S32, and the service path to be optimized is migrated to its target path at the optimization time, thereby completing the path migration. This path migration also releases the network node, and the next path migration is performed according to the above scheme based on the network node released this time, and so on, until the remaining service paths to be optimized in the resource interlocking optimization scenario complete the path migration.

[0130] Step S34, determining the optimization time of the unlock path, and migrating the rerouting path to the corresponding target path according to the optimization time of the unlock path.

[0131] After all other service paths to be optimized in the resource interlocking optimization scenario except the unlocked path have completed path migration, the optimization time of the unlocked path is determined, and the rerouting path is migrated to the corresponding target path according to the optimization time of the unlocked path.

[0132] In some embodiments, the step of determining the time of the service path to be optimized includes: predicting the traffic of the service to be optimized; determining the tidal time of the service to be optimized according to the traffic of the service to be optimized; and determining the optimization time of the service to be optimized according to the tidal time. In the resource interlocking optimization scenario, the optimization time of the service path to be optimized for this path migration is determined according to the tidal time of the service path to be optimized for this path migration. That is to say, in the process of path migration, after determining the service path to be optimized that needs to be migrated, the optimization time of the service path to be optimized is determined according to the tidal time of the corresponding service. It should be noted that the tidal time refers to the time when the service traffic is low, and the tidal time period of the service usually occurs periodically. For example, if the traffic of a certain service is low from 23:00 to 5:00 am the next day, the tidal time of the service can be selected from the time period from 23:00 to 5:00 am the next day. The optimization time is the exact time determined according to the tidal time. For example, it can be 23:30 on xxx year xxx month xxx day.

[0133] In order to ensure that path migration can proceed normally and is not affected by resource conflicts, in some embodiments, the time interval between the optimization times of two adjacent optimized business paths to be optimized is greater than a preset threshold. That is to say, the time interval between the optimization times of two optimized business paths to be optimized that perform path migration successively is greater than the preset threshold, and illustratively, the time interval is greater than or equal to 1 day. It should be noted that in the resource interlocking optimization scenario, the time interval between the rerouting time and the optimization time of the business path to be optimized for the first path migration is greater than the preset threshold, and the time interval between the optimization times of any other two business paths to be optimized for those path migrations is greater than the preset threshold.

[0134] To clearly illustrate the solution of the embodiment of the present disclosure, the following Figure 6 , Fig. 9 and Fig.10 , describes in detail the process of path migration in the resource interlocking optimization scenario. Figure 6 The three OTN service paths Path1, Path2, and Path3 shown by the solid lines are migrated to the three OTN service paths Path1', Path2', and Path3' shown by the dotted lines, which have the best global energy efficiency. Fig. 9 As shown in FIG. 1 , the process of path migration in the resource interlocking optimization scenario includes the following steps:

[0135] Step 1: Reroute the OTN service path Path2 from the source path AJL to the temporary path ADL, so that the network node J occupied by the source path of the OTN service path Path2 is released, thereby resolving the resource interlocking problem encountered by Path1, Path2, and Path3 during the process of optimizing the energy efficiency of each path.

[0136] Step 2: Migrate the OTN service path Path1 from the source path ABCJ to the target path AJ to obtain the path Path1' with the best global energy efficiency for the OTN service, and release the network nodes B and C.

[0137] Step 3: Migrate the OTN service path Path3 from the source path ALMJ to the target path ABJ to obtain the path Path3' with the best global energy efficiency for the OTN service, and release the network nodes L and M.

[0138] Step 4: Migrate the OTN service path Path2 from the temporary path ADL to the target path AL to obtain the path Path2' with the best global energy efficiency for the OTN service.

[0139] Corresponding to the above four operation steps, Fig.10 The implementation time point of each step (rerouting time, optimization time) is shown in the figure. These implementation time points are obtained based on the operation sequence of the above four steps and the use of time series AI algorithms such as Informer to predict the ebb and flow time of each OTN service flow. The time interval between the implementation time points of two adjacent steps is greater than 1 day.

[0140] In some embodiments, Fig.11 As shown, for the resource independent optimization scenario, optimizing each service path to be optimized in the optimization scenario one by one includes the following steps:

[0141] Step S41, determining the optimization time of each service path to be optimized in the resource independent optimization scenario.

[0142] Step S42: Migrate each service path to be optimized in the resource independent optimization scenario to a corresponding target path one by one according to the optimization time.

[0143] In some embodiments, the step of determining the time of the service path to be optimized includes: predicting the traffic of the service to be optimized; determining the tidal time of the service to be optimized based on the traffic of the service to be optimized; and determining the optimization time of the service to be optimized based on the tidal time. In the resource independent optimization scenario, the migration order is determined, and the optimization time of each service to be optimized is determined according to the migration order. Since there is no resource interlocking, the migration order of each service to be optimized is not limited and can be randomly generated. In other words, after determining the tidal time of each service to be optimized, the optimization time of each service to be optimized is determined according to the tidal time of each service to be optimized according to the migration order.

[0144] In order to ensure that the path migration can proceed normally and is not affected by resource conflicts, in some embodiments, the time interval between the optimization times of two adjacent optimized service paths to be optimized is greater than a preset threshold. In other words, the time interval between the optimization times of two optimized service paths to be optimized that undergo path migration successively is greater than a preset threshold, that is, in a resource independent optimization scenario, the time interval between the optimization times of any two service paths to be optimized is greater than the preset threshold, and illustratively, the time interval is greater than or equal to 1 day.

[0145] To clearly illustrate the solution of the embodiment of the present disclosure, the following Figure 7 , Fig.12 and Fig.13 , describes in detail the process of path migration in the resource independent optimization scenario. Figure 7 The three OTN service paths Path1, Path2, and Path3 shown by the solid lines are migrated to the three OTN service paths Path1', Path2', and Path3' shown by the dotted lines, which have the best global energy efficiency. Fig.12 As shown, the process of path migration in the resource independent optimization scenario includes the following steps:

[0146] Step 1: Migrate the OTN service path Path1 from the source path ABCJ to the target path AJ.

[0147] Step 2: Migrate the OTN service path Path2 from the source path ADL to the target path AL.

[0148] Step 3: Migrate the OTN service path Path3 from the source path JML to the target path JL.

[0149] Corresponding to the above four operation steps, Fig.13The implementation time point of each step (i.e., optimization time) is shown in the figure. These implementation time points are obtained based on the operation sequence of the above four steps and the use of time series AI algorithms such as Informer to predict the tidal time of each OTN service flow. The time interval between the implementation time points of two adjacent steps is greater than 1 day.

[0150] Fig.14 A schematic diagram of an OTN network energy efficiency optimization process provided by a specific example of the present disclosure is as follows: Fig.14 As shown, the network energy efficiency optimization process includes the following steps:

[0151] Step S51, detecting the energy consumption of W existing OTN services in the network topology and the creation / deletion of OTN services.

[0152] Step S52, determining whether there is any new OTN service being created or deleted, if M services are being created or K services are being deleted, executing step S53; if there is no new OTN service being created or deleted, executing step S54.

[0153] Step S53 , periodically taking (W+MK) OTN services as inputs of the pre-trained DRL&MCTS model, and inferring to obtain the path distribution with the best global energy efficiency for the (W+MK) OTN services.

[0154] The number of OTN services to be optimized is (W+MK).

[0155] Step S54, regularly taking W OTN services as input of the pre-trained DRL&MCTS model, and inferring to obtain the path distribution with the best global energy efficiency for the W OTN services.

[0156] The number of OTN services to be optimized is W.

[0157] Step S55, according to the current path distribution of the OTN service to be optimized and the path distribution with the best global energy efficiency, formulate a progressive path-by-path energy efficiency optimization plan, decompose the entire optimization process into several resource interlocking optimization scenarios and resource independent optimization scenarios, and implement progressive path-by-path energy efficiency optimization processing for these scenarios in parallel.

[0158] Step S56, determine whether the network operation life cycle has ended. If not, execute step S51; otherwise, end the process.

[0159] The disclosed embodiments relate to the application field of digital twin modeling in the core capability field of "energy efficiency problem identification and optimization" of self-intelligent OTN, and provide a method mechanism for realizing OTN network energy efficiency problem identification and optimization by using digital twin technology in a self-intelligent OTN network scenario. The network energy efficiency optimization method has minimal impact on the existing network operation and is unaware of users.

[0160] The embodiment of the present disclosure also provides a computer device, including: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by the one or more processors, the one or more processors implement the network energy efficiency optimization method as described above.

[0161] The embodiment of the present disclosure further provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed, the network energy efficiency optimization method as described above is implemented.

[0162] It will be appreciated by those skilled in the art that all or some of the steps in the method disclosed above, the functional modules / units in the device may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0163] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A network energy efficiency optimization method, characterized in that: include: Determine the action strategies of multiple services to be optimized according to a pre-trained action strategy learning model; wherein the action strategy learning model is a digital twin deep reinforcement learning network model, and the energy consumption optimization objective function of the action strategy learning model is to minimize the sum of the workload energy consumption and the equipment energy consumption of the multiple services to be optimized; the actual total reward of a trajectory of the action strategy learning model is the timely reward in the establishment state of the last service in the trajectory; According to the action strategy, traverse all service establishment states, determine the service path establishment action with the best global energy efficiency in each service establishment state, and obtain the service path establishment action sequence with the best global energy efficiency; An action sequence is established according to the service path with the best global energy efficiency, and the path distribution with the best global energy efficiency of the multiple services to be optimized is determined.

2. The method according to claim 1, characterized in that According to the action strategy, all service establishment states are traversed to determine the service path establishment action with the best global energy efficiency under the service establishment state, including: For each service establishment state, a Monte Carlo tree search algorithm is used to calculate the action scores corresponding to each alternative service path establishment action in the service establishment state according to the score of executing the alternative service path establishment action, the strategy score of the action strategy in the service establishment state, the number of times the alternative service path establishment action is selected in the service establishment state, and the preset hyperparameters; Determine a target action corresponding to the maximum value of the action score, execute the target action, obtain a next service establishment state, and sequentially generate subsequent service establishment states from the next service establishment state according to the action strategy until the trajectory ends; Calculating a state score of the next business establishment state according to a state value of the next business establishment state and a timely reward obtained from the next business establishment state to the end of the trajectory; According to executing the alternative service path establishment action in the service establishment state, a value of the state score record of the next service establishment state and each state score corresponding to the state score record in the next service establishment state are obtained, an average state score of the next service establishment state is calculated, and the score of executing the alternative service path establishment action is updated by using the average score; After traversing all service establishment states for a preset number of times, the service path establishment action with the best global energy efficiency in the service establishment state is determined according to the number of times each candidate service path establishment action is selected in the service establishment state.

3. The method according to claim 1, characterized in that After establishing an action sequence according to the service path with the best global energy efficiency and determining the distribution of the paths with the best global energy efficiency of the multiple services to be optimized, the method further includes: Determining multiple service paths to be optimized according to the distribution of paths with the best global energy efficiency of the multiple services to be optimized; Each of the service paths to be optimized is optimized one by one.

4. The method according to claim 3, characterized in that The optimizing each of the service paths to be optimized one by one includes: Dividing the service path to be optimized into optimization scenarios, where the types of the optimization scenarios include resource interlocking optimization scenarios and resource independent optimization scenarios; For each optimization scenario, each service path to be optimized in the optimization scenario is optimized one by one.

5. The method according to claim 4, characterized in that The type of the optimization scenario is a resource interlocking optimization scenario. For the resource interlocking optimization scenario, each of the service paths to be optimized in the optimization scenario is optimized one by one, including: Determine an unlocking path and a rerouting path of the unlocking path according to the resource interlocking state, wherein the unlocking path is one of the service paths to be optimized in the resource interlocking optimization scenario; Determining a rerouting time of the unlocking path, and migrating the unlocking path to the rerouting path according to the rerouting time to release a network node; Determine the service path to be optimized for this path migration and the optimization time of the service path to be optimized according to the network nodes released by the previous path migration, and migrate the service path to be optimized for this path migration to the corresponding target path according to the optimization time until all the remaining service paths to be optimized in the resource interlocking optimization scenario complete the path migration; An optimization time of the unlock path is determined, and according to the optimization time of the unlock path, the rerouting path is migrated to a corresponding target path.

6. The method according to claim 4, characterized in that The type of the optimization scenario is a resource-independent optimization scenario. For the resource-independent optimization scenario, each of the service paths to be optimized in the optimization scenario is optimized one by one, including: Determining the optimization time of each of the service paths to be optimized in the resource independent optimization scenario; According to the optimization time, each of the service paths to be optimized in the resource independent optimization scenario is migrated to a corresponding target path one by one.

7. The method according to claim 5 or 6, characterized in that The step of determining the time of the service path to be optimized includes: Predicting the traffic of the service to be optimized; Determining the ebb and flow time of the service to be optimized according to the traffic of the service to be optimized; The optimization time of the business to be optimized is determined according to the tidal fall time; wherein, in the resource interlocking optimization scenario, the optimization time of the business path to be optimized of this path migration is determined according to the tidal fall time of the business path to be optimized of this path migration; in the resource independent optimization scenario, the migration order is determined, and the optimization time of each business to be optimized is determined according to the migration order.

8. The method according to claim 5 or 6, characterized in that The time interval between two adjacent optimization times of the service path to be optimized is greater than a preset threshold.

9. A computer device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the network energy efficiency optimization method according to any one of claims 1 to 8.

10. A computer readable medium having a computer program stored thereon, wherein: When the program is executed, the network energy efficiency optimization method as described in any one of claims 1 to 8 is implemented.

Images