Network energy efficiency optimization method, computer device, and computer-readable storage medium

Abstract

Provided in the present disclosure is a network energy efficiency optimization method. The method comprises: using a digital twin reinforcement learning network model for learning, so as to obtain an action policy for a plurality of services to be optimized; traversing all service establishment states according to the action policy, and determining service path establishment actions having optimal global energy efficiency in the service establishment states, so as to obtain a sequence of service path establishment actions having the optimal global energy efficiency; and according to the sequence of service path establishment actions having the optimal global energy efficiency, determining a path distribution having the optimal energy efficiency for said services. Further provided in the present disclosure are a computer device and a computer-readable storage medium.

Description

Network energy efficiency optimization method, computer device and computer readable storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202310906270.8 filed on July 21, 2023, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to the field of self-intelligent optical network technology, and in particular to a network energy efficiency optimization method, a computer device, and a computer-readable storage medium. Background Art

[0004] The global communications industry is moving from the interconnected and cloud eras to the intelligent era. Faced with market competition, industry-wide digital transformation, and increasing network complexity, which present challenges in network planning, optimization, operations and maintenance, reliability, agility, and OPEX reduction, the evolution of optical networks toward intelligence is inevitable. The emergence and rise of intelligent optical network technology provides a solid foundation and primary means for achieving this goal.

[0005] Public content

[0006] On the one hand, an embodiment of the present disclosure provides a network energy efficiency optimization method, including: determining the action strategies of multiple services to be optimized based on a pre-trained action strategy learning model; the action strategy learning model is a digital twin deep reinforcement learning network model, and the energy consumption optimization objective function of the action strategy learning model is to minimize the sum of the workload energy consumption and equipment energy consumption of the multiple services to be optimized; the actual total reward of a trajectory of the action strategy learning model is the timely reward under the last service establishment state in the trajectory; according to the action strategy, traverse all service establishment states, determine the service path establishment action with the best global energy efficiency under each of the service establishment states, and obtain the service path establishment action sequence with the best global energy efficiency; according to the service path establishment action sequence with the best global energy efficiency, determine the path distribution with the best global energy efficiency for the multiple services to be optimized.

[0007] On the other hand, an embodiment of the present disclosure also provides a computer device, comprising: at least one processor; a storage device on which at least one computer program is stored; the at least one computer program is executed by the at least one processor, so that the at least one processor implements the network energy efficiency optimization method as described above.

[0008] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor so that the processor implements the network energy efficiency optimization method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG1 is a flow chart of a method for optimizing network energy efficiency according to an embodiment of the present disclosure;

[0010] FIG2 is a schematic diagram of an action strategy learning model trained using a Q-Based Actor-Critic algorithm according to an embodiment of the present disclosure;

[0011] FIG3 is a schematic diagram of a process for determining a service path establishment action with the best global energy efficiency under a service establishment state according to an embodiment of the present disclosure;

[0012] 4a to 4e are schematic diagrams of determining a service path establishment action with the best global energy efficiency using the MCTS algorithm according to an embodiment of the present disclosure;

[0013] FIG5 is a flow chart of a method for optimizing network energy efficiency according to an embodiment of the present disclosure;

[0014] FIG6 is a schematic diagram of a resource interlocking optimization scenario provided by an embodiment of the present disclosure;

[0015] FIG7 is a schematic diagram of a resource independent optimization scenario provided by an embodiment of the present disclosure;

[0016] FIG8 is a schematic diagram of a process for optimizing a service path to be optimized in a resource interlocking optimization scenario according to an embodiment of the present disclosure;

[0017] FIG9 is a schematic diagram of optimizing a service path to be optimized in a resource interlocking optimization scenario provided by a specific example of the present disclosure;

[0018] FIG10 is a schematic diagram of the unlocking time and optimization time of each service path to be optimized in FIG9 ;

[0019] FIG11 is a schematic diagram of a process for optimizing a service path to be optimized in a resource independent optimization scenario according to an embodiment of the present disclosure;

[0020] FIG12 is a schematic diagram of optimizing a service path to be optimized in a resource independent optimization scenario provided by a specific example of the present disclosure;

[0021] FIG13 is a schematic diagram of the unlocking time and optimization time of each service path to be optimized in FIG12;

[0022] FIG14 is a flow chart of a network energy efficiency optimization method provided in a specific example of the present disclosure. DETAILED DESCRIPTION

[0023] Hereinafter, example embodiments will be described more fully with reference to the accompanying drawings, but the example embodiments may be embodied in different forms, and the present disclosure should not be construed as limited to the embodiments set forth herein. These embodiments are provided to make the present disclosure more thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0024] As used herein, the term "and / or" includes any and all combinations of at least one of the associated listed items.

[0025] The terms used herein are used only to describe specific embodiments and do not limit the present disclosure. As used herein, the singular forms "a," "an," and "the" also include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, they specify the presence of a particular feature, whole, step, operation, element, and / or component, but do not exclude the presence or addition of at least one other feature, whole, step, operation, element, component, and / or group thereof.

[0026] The embodiments described herein may be described with reference to plan views and / or cross-sectional views, with the aid of idealized schematic diagrams of the present disclosure. Therefore, the example illustrations may be modified based on manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings are schematic in nature, and the shapes of the regions shown in the drawings illustrate specific shapes of the regions of the components, but are not limiting.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0028] Autonomous Driving Network technology originates from the "Autonomous Driving Networks" (AN) concept proposed by TM Forum in 2019. Two years later, the AN concept has become a consensus within the industry. Its purpose is to enable the digital transformation of operator networks through the integration of network and digital technologies. This provides innovative ICT services and user experiences with zero wait, zero contact, and zero failures for vertical industries and consumers, while also building self-configuring, self-healing, and self-optimizing network capabilities throughout the operator network operation lifecycle. Consequently, the autonomous optical network technology, derived from this foundation, achieves intelligent management, a superior experience, and flexible, open intelligence through intelligent O&M capabilities like self-configuration, self-recovery, and self-optimization, meeting customer needs for improved user experience and reduced operating costs.

[0029] Among the many intelligent capabilities required of an intelligent optical network (AN OTN), identifying and optimizing energy efficiency issues is a key core capability for achieving OTN network intelligence. With global warming worsening, energy conservation, consumption reduction, and carbon emissions reduction have significant international economic and political significance.

[0030] At the same time, emerging digital twin (DT) technology is entering a period of rapid development and is being widely applied in manufacturing, supply chain, and other fields to support industrial digital transformation. Digital twin technology is also a strong driver of digital transformation in the telecommunications sector. The industry considers digital twin technology to be the foundation of digital transformation and a crucial component supporting the realization of autonomous network architecture and technologies. Network digital twins accurately perceive their own and external environments, creating digital mirrors of their internal and external environments. By integrating simulation with predictive and preventive capabilities, they play a key enabling role in scenarios such as "low-cost trial and error," "intelligent decision-making," and "predictive maintenance."

[0031] Digital twins, or DT technology, can be understood as building a digital (virtual) model of a physical object in a digital space, and using data from the physical object to continuously correct the model and update its state, so that the model remains consistent with the physical object throughout its life cycle and can accurately mirror the real-time operating state of the physical object. This model becomes the digital twin of the physical object (abbreviated as digital twin). Based on digital twins, monitoring, analysis, prediction, diagnosis, training, and simulation can be performed, and the simulation results can be fed back to the physical object to help optimize and make decisions about the physical object. Related technologies involving digital twin model construction, real-time updating of digital model states, simulation analysis based on digital twins, and control decisions can be collectively referred to as digital twin DT technology. It can be seen from this that how to combine the structural characteristics and state characteristics of the physical object being mirrored by the twin with the application scenarios of DT to construct a DT model is a key technology of DT.

[0032] In summary, how to use digital twin DT modeling and analysis technology to effectively identify low energy efficiency problems in OTN network operations and optimize them to improve energy efficiency, and how to ensure that energy efficiency optimization methods and measures have little impact on existing network operations and are imperceptible to users, has become a key issue of self-intelligence that the industry is currently paying attention to and needs to be solved.

[0033] The present disclosure provides a method and mechanism for identifying and optimizing OTN network energy efficiency issues using digital twin modeling technology. The method can be applied to OTN, PTN (Packet Transport Network), POTN (Packet Optical Transport Network), SPN (Slicing Packet Network), IP and other networks. The present disclosure uses OTN network as an example for illustration.

[0034] Assume that n OTN services are created under a given OTN network topology (including mesh and star topologies). In this disclosure, creating an OTN service refers to creating an OTN ODU (Optical Channel Data Unit) layer service, and that the OCH (Optical Channel with full functionality) channel resources of the network topology fully meet the requirements for creating n OTN services, where n > 1. The action strategy learning model for creating these n OTN services is defined as follows: 1 to 6.

[0035] 1. Define the OTN network optimization objective function

[0036] The objective function of OTN network optimization can be to minimize the sum of workload energy consumption and equipment energy consumption of the OTN service to be optimized. MinE , that is, the minimum energy consumption required by the entire network to support and meet the normal operation of the n OTN services within a specified time period, in kilowatt-hours. In this disclosure, the OTN network optimization objective function is the energy consumption optimization objective function, which is expressed as follows:

[0037] E(Svc i ) indicates that in the current trajectory of creating n services, the i-th OTN service Svc i The workload energy consumption is related to the energy efficiency ratio of the node / port that the i-th OTN service path passes through. Indicates the Topo of the entire network AThe energy consumption of the equipment required to support the normal operation of the n OTN services. For example, if a network node only carries one OTN service, and this service only occupies part of the time slot resources on the line-side OCH channel, but in order to ensure the normal operation of this service, the relevant equipment of the network node (main control board, etc.) needs to be powered on, and the OTU board and port corresponding to the OCH channel also need to be powered on. This consumes energy in addition to the energy required for the workload of the OTN service. Therefore, It can also be called extra energy consumption. A The total additional energy consumption of all n OTN services is

[0038] 2. Define the state

[0039] Status S t (t=0,…,n-1) indicates that t OTN services have been created (including detailed information of all created services), (nt) OTN services have not been created (including detailed information of all uncreated OTN services), and the (t+1)th OTN service is about to be created.

[0040] Status S n Indicates that the creation of n OTN services for the entire network is complete.

[0041] 3. Define the trajectory

[0042] Using a certain action strategy to complete the sequential creation of n OTN services across the entire network is defined as a trajectory.

[0043] 4. Define immediate rewards and final return mechanisms

[0044] R t (t=0,…,n-1) represents the state S t Taking "minimum energy consumption of the entire network" as the OTN network optimization objective function as an example, this paper adopts a delayed reward mechanism, R t The value is 0.

[0045] G represents the total reward obtained by the agent at the end of a trajectory: R n = G. The value of G can be defined as the reciprocal of the energy consumption value of the entire network service (excluding the energy consumption unit, kilowatt-hour) multiplied by 1000, that is, n represents the total number of OTN services in the entire network. Thus, the service distribution solution with lower energy consumption has higher total returns.

[0046] G t Indicates that in state St Next, perform action a t The expected return, γ is the discount coefficient, 0<γ<1.

[0047] 5. Define actions and action strategies

[0048] An action a t (t=0,…,n-1) means that in the current network topology environment, among the pending services, one is selected as the next service to be created, and one is selected as the resource route for the pending service from multiple alternative routes (routes to which network resources have been allocated, including latency, bandwidth, availability, energy consumption, and other resources) to complete the process of creating the pending service.

[0049] Multiple candidate resource routes for a proposed OTN service can be ranked based on the optimization strategy corresponding to the OTN network optimization objective function. For example, if the OTN network optimization objective function is to minimize network-wide energy consumption, the multiple candidate resource routes for the proposed OTN service can be ranked in ascending order based on energy consumption.

[0050] SvcID t+1 Indicates that in the current trajectory of creating n businesses, in state S t Next (t+1) OTN service Svc to be built t+1 The business ID (ID) of action a t Decide.

[0051] The action strategy π(s,a) (also written as π(s)) represents the probability of selecting a pending OTN service and, under the condition that the pending OTN service is selected, selecting a candidate resource route. It also represents the order in which the pending OTN services (including the routes of the pending services) are created.

[0052] 6. Define state behavior value function and state value function

[0053] The state behavior value function is expressed as: It represents the expected reward obtained after the agent starts from state S and performs action a according to the action strategy π(s,a).

[0054] The state value function is expressed as: Represents all state behavior values ​​Q under state S π The weighted sum of (s,a). π(a|s) represents the probability of executing action a in state S according to the action policy π(s,a).

[0055] FIG1 is a flow chart of a network energy efficiency optimization method provided by an embodiment of the present disclosure. As shown in FIG1 , the network energy efficiency optimization method includes the following steps S11 to S13 .

[0056] Step S11: determine the action strategies of multiple services to be optimized based on the pre-trained action strategy learning model; the action strategy learning model is a digital twin deep reinforcement learning network model, and the energy consumption optimization objective function of the action strategy learning model is to minimize the sum of the workload energy consumption and equipment energy consumption of the multiple services to be optimized; the actual total return of a trajectory of the action strategy learning model is the timely reward in the establishment state of the last service in the trajectory.

[0057] In this disclosure, the action strategy learning model is a deep reinforcement learning network model (DRL) generated using digital twin technology. The convergence function of this action strategy learning model is the energy consumption optimization objective function. The other parameter definitions of this action strategy learning model are as described above and will not be repeated here.

[0058] Step S12: traverse all service establishment states according to the action strategy, determine the service path establishment action with the best global energy efficiency in each service establishment state, and obtain the service path establishment action sequence with the best global energy efficiency.

[0059] In the entire network Topo A In a topological environment, the Monte Carlo Tree Search (MCTS) algorithm can be used to further fine-tune the action strategy learned in step S11 to obtain the globally energy-efficient service path establishment action for each service establishment state. Based on the globally energy-efficient service path establishment action for each service establishment state, a complete globally energy-efficient service path establishment action sequence can be obtained.

[0060] Step S13: establishing an action sequence according to the service path with the best global energy efficiency, and determining the distribution of multiple paths with the best global energy efficiency for the services to be optimized.

[0061] By establishing an action sequence based on the business path with the best global energy efficiency, the global energy efficiency optimal distribution of n businesses to be optimized can be obtained.

[0062] The network energy efficiency optimization method provided by the present disclosure utilizes a digital twin reinforcement learning network model to learn action strategies for multiple services to be optimized. The energy consumption optimization objective function of the action strategy learning model is to minimize the sum of the workload energy consumption and the equipment energy consumption of each service to be optimized, and the action strategy learning model adopts a delayed reward mechanism; according to the action strategy, all service establishment states are traversed, and the service path establishment action with the best global energy efficiency under each service establishment state is determined to obtain a service path establishment action sequence with the best global energy efficiency; based on the service path establishment action sequence with the best global energy efficiency, the path distribution with the best global energy efficiency for each service to be optimized is determined; the present disclosure utilizes digital twin modeling and analysis technology to effectively identify low energy efficiency problems occurring in network operations, and determine the path distribution with the best global energy efficiency for each service, so as to configure the path with the best global energy efficiency for each service, thereby optimizing and improving energy efficiency.

[0063] In the entire network Topo A In the topology environment, the Actor-Critic algorithm can be used to obtain the optimal global energy efficiency distribution Q for selecting n services. ω (s,a) state-value neural network and π θ (s,a) Action strategy neural network, using Actor-Critic algorithm modeling and model processing flow is as follows:

[0064] According to the design in the Actor algorithm, the parameterized action strategy function is defined as π θ (s,a),π θ The value of (s,a) can be obtained by the trained π used in the Actor algorithm. θ The value is inferred by the deep neural network, and θ is the parameter vector of the neural network layer.

[0065] According to the design of the Critic algorithm, the parameterized state behavior value function is defined as Q ω (s,a)=φ(s,a) T ω, Q ω The value of (s,a) can be inferred through the trained Q-value deep neural network used for the Critic algorithm. ω is the parameter vector of the neural network layer, and φ(s,a) is the feature vector function of state S and action a, which can be used as the input of the Critic deep neural network.

[0066] Figure 2 is a schematic diagram of an action strategy learning model trained using the Q-Based Actor-Critic (QAC) algorithm, as provided in an embodiment of the present disclosure. The Q-Based Actor-Critic algorithm is a typical DRL algorithm. As shown in Figure 2, the process of using the QAC algorithm in conjunction with the action strategy learning model to determine the action strategy for the service to be optimized is described below.

[0067] Initialize the entire network topology environment, including initializing s0∈S and action strategy parameter vector θ.

[0068] Step A1: Initialize sampling actions a~π according to the action strategy θ .

[0069] Step A2, calculate the score Q of action a ω (s,a),Q ω (s,a)=φ(s,a) T ω.

[0070] For each step of the sampling action, execute the following steps A3 to A9.

[0071] Step A3, sampling and timely reward Sample the next state transition

[0072] Step A4: Sample the next action a'~π according to the action strategy θ (s',a').

[0073] Step A5, calculate the score Q of action a' ω (s',a'),Q ω (s',a')=φ(s',a') T ω.

[0074] Step A6, calculate the intermediate value δ, δ = r + γQ ω (s',a')-Q ω (s,a).

[0075] Step A7, determine the action strategy parameter vector θ, The action strategy parameter vector θ is the Actor model parameter.

[0076] Step A8: Determine ω←ω+βδφ(s,a) based on the intermediate value δ and the constant β, where ω is a parameter of the Critic model.

[0077] Step A9, update the action and state, a←a', s←s'.

[0078] End for.

[0079] End processing.

[0080] In some implementations, a Monte Carlo tree search algorithm is used to traverse all service establishment states according to the action strategy to determine the service path establishment action with the best global energy efficiency for each service establishment state. As shown in FIG3 , for each service establishment state, a Monte Carlo tree search algorithm is used to traverse all service establishment states according to the action strategy to determine the service path establishment action with the best global energy efficiency for that service establishment state, including the following steps S21 to S25.

[0081] Step S21, calculate the action score corresponding to each alternative business path establishment action in the business establishment state according to the score of executing the alternative business path establishment action, the policy score of the action policy in the business establishment state, the number of times the alternative business path establishment action is selected in the business establishment state and the preset hyperparameters.

[0082] A service establishment state St may correspond to at least one alternative service path establishment action. The next service establishment state S obtained by executing different alternative service path establishment actions under the service establishment state St is t+1 To select the service path with the best overall energy efficiency, the most appropriate service path establishment action must be selected from the various candidate service path establishment actions in each service establishment state. In this step, each candidate service path establishment action is scored to obtain a corresponding action score.

[0083] Step S22: determine the target action corresponding to the maximum action score, execute the target action, obtain the next service establishment state, and generate subsequent service establishment states from the next service establishment state according to the action strategy until the trajectory ends.

[0084] The alternative service path establishment action with the highest action score is selected from all the alternative service path establishment actions in the service establishment state St as the execution action for the next expansion, and the selected alternative service path establishment action is executed to obtain the next service establishment state S t+1 By analogy, each subsequent business establishment state is generated step by step until the business establishment state S is generated. n , until the current trajectory ends.

[0085] Step S23 , calculating the state score of the next service establishment state according to the state value of the next service establishment state and the timely reward obtained from the next service establishment state to the end of the trajectory.

[0086] Step S24, based on executing the alternative service path establishment action in the service establishment state, obtain the value of the status score record of the next service establishment state and the status scores corresponding to the status score records in the next service establishment state, calculate the average status score of the next service establishment state, and use the average status score to update the score of executing the alternative service path establishment action.

[0087] This step performs backtracking operations and executes multiple actions a in the business establishment state St t Multiple evaluation simulations are performed to obtain corresponding multiple state scores, thereby generating a state score record, which includes multiple state scores. The value L of the state score record is the value of executing the alternative service path establishment action a under the service establishment state St. t In this step, the average state score is calculated based on the sum of the state scores and the value L of the state score record, and the average state score is used to update the execution of the alternative service path establishment action a. t score.

[0088] Step S25 , after traversing all service establishment states for a preset number of times, the service path establishment action with the best global energy efficiency in the service establishment state is determined according to the number of times each candidate service path establishment action is selected in the service establishment state.

[0089] After a preset number of Monte Carlo tree search processes, the alternative service path establishment actions a under the service establishment state St are obtained. t The number of times N(a t ), select N(a t ) The maximum value of the alternative service path establishment action a t , as the alternative service path establishment action that the Agent needs to select in the service establishment state St. The preset number of times can be determined according to the needs of the algorithm application.

[0090] Figures 4a to 4e are schematic diagrams of determining the global energy-efficient service path establishment action using the MCTS algorithm provided in an embodiment of the present disclosure. The following describes in detail the process of determining the global energy-efficient service path establishment action using the MCTS algorithm in conjunction with Figures 4a to 4e.

[0091] As shown in Figure 4a, there are three alternative service path establishment actions in the service establishment state St, namely a t =a1, a t =a2, a t =a3, and the action scores corresponding to the above three alternative service path establishment actions are calculated respectively according to the following formula (1):

[0092] U score (at ) is the action score corresponding to the action of establishing the alternative service path; Q(a t ) establishes action a for executing alternative business paths obtained through MCTS search t score; η is a preset hyperparameter that can be adjusted manually; π(a t |s t ) is the action strategy π(s t ) In the service establishment state St, the corresponding alternative service path establishment action a t Strategy score; N(a t ) Select an alternative service path to establish action a for the service establishment state St t times.

[0093] According to the above formula (1), the action scores corresponding to the above three alternative service path establishment actions are calculated respectively, U score (a1)=200,U score (a2) = 300, U score (a3)=250.

[0094] As shown in FIG4b, the alternative service path establishment action a2 with the highest action score is selected as the next extended execution action, and the next service establishment state S is generated accordingly. t+1 .

[0095] As shown in Figure 4c, the action strategy π(s t ), from the next business establishment state S t+1 Start the expansion execution (Fast Rollout) until the business establishment status S is generated n The next service establishment state S is calculated according to the following formula (2): t+1 Status score:

[0096] V score (s t+1 ) Establish state S for the next business t+1 Status score of V π (s t+1 ) is the action strategy π(s t ) in the case of the next business establishment state S t+1 Status value of R n To adopt the action strategy π(s t ), from the next business establishment state S t+1 To the end of the track (ie generate business establishment state S n ) when the instant reward is obtained, R n =G.

[0097] As shown in Figure 4d, the backtracking operation is performed to execute the alternative service path establishment action a. t , perform multiple evaluation simulations to establish the state S from the next business t+1 Expand each V score(l) (S t+1 ) to perform cumulative averaging, calculate and update the alternative action a t Q(a t ) value. Calculate the next service establishment state S according to the following formula (3): t+1 The average state score and update the alternative action a t Q(a t ):

[0098] Q(a t ) as the execution alternative service path establishment action a t Score, establish state S for the next business t+1 The average state score of L is the execution of the alternative service path establishment action a under the service establishment state St t Get the next service establishment status S t+1 The value of the status score record.

[0099] As shown in Figure 4e, after a preset number of Monte Carlo tree search processes, the action selection decision of the Agent in the service establishment state St follows the following principles: a t =argmax N(a t ), N(a t ) is the establishment action a of each alternative service path under the service establishment state St t The number of times selected. That is, select N(a t ) value is the alternative service path establishment action that the Agent needs to select in the service establishment state St.

[0100] Based on the above steps, the MCTS algorithm can be used for each service establishment state to obtain the service path establishment action that needs to be selected in each service establishment state, that is, in the entire network Topo A In a topological environment, the service path establishment action with the best global energy efficiency under each service establishment state is determined, thereby obtaining the path distribution with the best global energy efficiency for n services to be optimized.

[0101] In some embodiments, as shown in FIG5 , after establishing an action sequence based on the service path with the best global energy efficiency and determining the path distribution with the best global energy efficiency for the multiple services to be optimized (i.e., step S13), the network energy efficiency optimization method may further include the following steps S14 and S15.

[0102] Step S14: Determine multiple service paths to be optimized based on the distribution of paths with optimal global energy efficiency for the multiple services to be optimized.

[0103] Step S15: Optimize each service path to be optimized one by one.

[0104] In some implementations, the step of optimizing each of the service paths to be optimized one by one (ie, step S15 ) includes the following steps S151 and S152 .

[0105] Step S151 : Divide the service path to be optimized into optimization scenarios, where the types of optimization scenarios include resource interlocking optimization scenarios and resource independent optimization scenarios.

[0106] The service paths to be optimized can be divided into multiple different optimization scenarios. Specifically, the service paths to be optimized can be divided into resource-interlocked optimization scenarios and resource-independent optimization scenarios. The OTN service paths to be optimized in each optimization scenario are optimized for each optimization scenario. It should be noted that the service paths to be optimized can also be divided into a single optimization scenario. In this case, each service path to be optimized in that optimization scenario is optimized.

[0107] Figure 6 is a schematic diagram of a resource interlocking optimization scenario provided by an embodiment of the present disclosure. As shown in Figure 6, the OTN network topology contains three OTN service paths: Path1, Path2, and Path3, represented by the three solid lines in Figure 6. The DRL & MCTS algorithm determines the optimal path distribution for these three OTN services, as shown by the three dashed lines in Figure 6. There is a resource occupation conflict between the source path of Path2 and the optimized target path Path1' after Path1 on the AJ link segment; there is a resource occupation conflict between the source path of Path3 and the optimized target path Path2' after Path2 on the AL link segment; and there is a resource occupation conflict between the source path of Path1 and the optimized target path Path3' after Path3 on the AB link segment.

[0108] Figure 7 is a schematic diagram of an independent resource optimization scenario provided by an embodiment of the present disclosure. As shown in Figure 7, the OTN network topology contains three OTN service paths: Path 1, Path 2, and Path 3, represented by the three solid lines in Figure 7. The DRL & MCTS algorithm determines the optimal path distribution for these three OTN services, which is represented by the three dashed lines in Figure 7. During the progressive, path-by-path optimization process from source to target, these three OTN services avoid resource conflicts. Each OTN service's resource optimization process is relatively independent and unconstrained by the resource usage of other OTN service paths.

[0109] Step S152 : For each optimization scenario, optimize each service path to be optimized in the optimization scenario one by one.

[0110] For each type of optimization scenario, an optimization strategy matching the type of the optimization scenario is selected to perform path migration, thereby migrating each service path to be optimized to a corresponding target path to achieve energy efficiency optimization.

[0111] It can be seen from the above steps S151 and S152 that in the network energy efficiency optimization method provided by the present invention, the DRL&MCTS algorithm technology can be used to obtain W business energy efficiency optimization target paths in the specified network topology; according to the progressive optimization scheme, the W business energy efficiency optimization process is decomposed into two resource optimization scenarios, namely, the resource interlocking optimization scenario and the resource independent optimization scenario, the number of resource interlocking optimization scenarios is m, m is greater than or equal to 1, and the number of resource independent optimization scenarios is n, n is greater than or equal to 1; progressive energy efficiency optimization processing is performed on the business paths in the m resource interlocking optimization scenarios and the n resource independent optimization scenarios respectively, so that progressive energy efficiency optimization can be performed on batch business paths in the specified network topology.

[0112] Due to the problem of resource interlocking, it is impossible to optimize each service path to be optimized one by one. Therefore, this disclosure proposes a method mechanism for gradually optimizing the energy efficiency of each service by rerouting a service path to be optimized to a temporary path to release the resource interlocking.

[0113] In some implementations, as shown in FIG8 , for a resource interlocking optimization scenario, optimizing each service path to be optimized in the optimization scenario one by one (step S152 ) includes the following steps S31 to S34 .

[0114] Step S31 : determining an unlocking path and a rerouting path of the unlocking path according to the resource interlocking state.

[0115] The unlocking path is one of the business paths to be optimized in the resource interlocking optimization scenario. According to the resource interlocking status, a business path to be optimized (i.e., the source path) is found for unlocking the resource interlocking status, and a rerouting path for the business path to be optimized is determined. The starting end node and the ending end node of the rerouting path correspond to the starting end node and the ending end node of the unlocking path, respectively.

[0116] Step S32: determining the rerouting time of the unlocked path, and migrating the unlocked path to the rerouting path according to the rerouting time to release the network node.

[0117] In this step, the migration of the unlocked path is completed, and the network nodes occupied by the unlocked path and required to be occupied by the target paths of other service paths to be optimized are correspondingly released.

[0118] In step S33, based on the network nodes released by the previous path migration, the service path to be optimized for this path migration and the optimization time of the service path to be optimized are determined, and the service path to be optimized for this path migration is migrated to the corresponding target path according to the optimization time until all the remaining service paths to be optimized in the resource interlocking optimization scenario complete the path migration.

[0119] After the unlocked path migration is complete, the service path to be optimized and its optimization time are determined based on the network nodes released in step S32. The service path to be optimized is then migrated to its target path at the optimization time, completing the path migration. This path migration also releases network nodes, and the next path migration is performed based on the released network nodes according to step S33, and so on, until all remaining service paths to be optimized in the resource interlocking optimization scenario have completed path migration.

[0120] Step S34 : determining the optimization time of the unlock path, and migrating the rerouting path to the corresponding target path according to the optimization time of the unlock path.

[0121] After all other service paths to be optimized in the resource interlock optimization scenario except the unlocked path have completed path migration, the optimization time of the unlocked path is determined, and the rerouted path is migrated to the corresponding target path according to the optimization time of the unlocked path.

[0122] In some embodiments, determining the optimization time of the business path to be optimized includes: predicting the traffic of the business to be optimized; determining the settling time of the business to be optimized based on the traffic of the business to be optimized; and determining the optimization time of the business to be optimized based on the settling time. In the resource interlocking optimization scenario, the optimization time of the business path to be optimized for this path migration is determined based on the settling time of the business path to be optimized for this path migration. That is to say, during the path migration process, after determining the business path to be optimized that currently needs to be migrated, the optimization time of the business path to be optimized is determined based on the settling time of the corresponding business. It should be noted that the settling time refers to the time when the business traffic is low. Usually, the settling time of the business occurs periodically. For example, if the traffic of a certain business is low from 23:00 to 5:00 am the next day, the settling time of the business can be selected from the time period from 23:00 to 5:00 am the next day. The optimization time is the exact time determined based on the settling time. For example, it can be 23:30 on xxx / xxx / xxx.

[0123] In order to ensure that path migration can proceed normally and is not affected by resource conflicts, in some embodiments, the time interval between the optimization times of two adjacent optimized business paths to be optimized is greater than a preset threshold. That is, the time interval between the optimization times of two optimized business paths to be optimized that undergo path migration successively is greater than the preset threshold. Exemplarily, the time interval is greater than or equal to 1 day. It should be noted that in the resource interlocking optimization scenario, the time interval between the rerouting time and the optimization time of the business path to be optimized that undergoes path migration for the first time is greater than the preset threshold, and the time interval between the optimization times of any other two business paths to be optimized that undergo path migration is greater than the preset threshold.

[0124] To clearly illustrate the network energy efficiency optimization method provided by this disclosure, the following detailed description of the path migration process in the resource interlocking optimization scenario is provided in conjunction with Figures 6, 9, and 10. The three OTN service paths Path1, Path2, and Path3, shown as solid lines in Figure 6, are migrated to the three OTN service paths Path1', Path2', and Path3', shown as dashed lines, that achieve the best global energy efficiency. As shown in Figure 9, the path migration process in the resource interlocking optimization scenario includes steps B1 through B4.

[0125] In step B1, the OTN service path Path2 is rerouted from the source path AJL to the temporary path ADL. This frees up the network node J occupied by the source path of the OTN service path Path2, thereby resolving the resource interlocking problem encountered during the energy efficiency optimization of each path among Path1, Path2, and Path3.

[0126] In step B2, the OTN service path Path1 is migrated from the source path ABCJ to the target path AJ to obtain the OTN service path Path1' with the best global energy efficiency, and network nodes B and C are released.

[0127] In step B3, the OTN service path Path3 is migrated from the source path ALMJ to the target path ABJ to obtain the OTN service path Path3' with the best global energy efficiency, and the network nodes L and M are released.

[0128] Step B4: Migrate the OTN service path Path2 from the temporary path ADL to the target path AL to obtain the OTN service path Path2' with the best global energy efficiency.

[0129] Corresponding to the above four operation steps, Figure 10 shows the implementation time points (rerouting time, optimization time) of each step. These implementation time points are obtained based on the operation sequence of the above four steps and the use of time series AI algorithms such as Informer to predict the ebb and flow time of each OTN service traffic. The time interval between the implementation time points of two adjacent steps is greater than 1 day.

[0130] In some implementations, as shown in FIG11 , for a resource-independent optimization scenario, optimizing each service path to be optimized in the optimization scenario one by one (step S152 ) includes the following steps S41 and S42 .

[0131] Step S41: Determine the optimization time of each service path to be optimized in the resource independent optimization scenario.

[0132] Step S42 : Migrating each service path to be optimized in the independent resource optimization scenario to a corresponding target path one by one according to the optimization time.

[0133] In some embodiments, determining the optimization time of the service path to be optimized (step S41) includes: predicting the traffic of the service to be optimized; determining the settling time of the service to be optimized based on the traffic of the service to be optimized; and determining the optimization time of the service to be optimized based on the settling time. In the resource-independent optimization scenario, the migration order is determined, and the optimization time of each service to be optimized is determined based on the migration order. Since there is no resource interlocking, the migration order of each service to be optimized is not limited and can be randomly generated. That is to say, after determining the settling time of each service to be optimized, the optimization time of each service to be optimized is determined according to the migration order based on the settling time of each service to be optimized.

[0134] To ensure that path migration proceeds normally and is not affected by resource conflicts, in some implementations, the time interval between the optimization times of two adjacent service paths to be optimized is greater than a preset threshold. In other words, the time interval between the optimization times of two service paths to be optimized that undergo path migrations is greater than a preset threshold. That is, in a resource-independent optimization scenario, the time interval between the optimization times of any two service paths to be optimized is greater than a preset threshold. Exemplarily, this time interval is greater than or equal to one day.

[0135] To clearly illustrate the network energy efficiency optimization method provided by this disclosure, the following detailed description of the path migration process in a resource-independent optimization scenario is provided in conjunction with Figures 7, 12, and 13. The three OTN service paths Path1, Path2, and Path3, shown as solid lines in Figure 7, are migrated to the three OTN service paths Path1', Path2', and Path3', shown as dashed lines, that achieve the best global energy efficiency. As shown in Figure 12, the path migration process in a resource-independent optimization scenario includes the following steps: C1 to C3.

[0136] Step C1: Migrate the OTN service path Path1 from the source path ABCJ to the target path AJ.

[0137] Step C2: Migrate the OTN service path Path2 from the source path ADL to the target path AL.

[0138] Step C3: Migrate the OTN service path Path3 from the source path JML to the target path JL.

[0139] Corresponding to the above three operation steps, Figure 13 shows the implementation time point (i.e., optimization time) of each step. These implementation time points are obtained based on the operation sequence of the above three steps and the use of time series AI algorithms such as Informer to predict the ebb and flow time of each OTN service traffic. The time interval between the implementation time points of two adjacent steps is greater than 1 day.

[0140] FIG14 is a flow chart of an OTN network energy efficiency optimization method provided by a specific example of the present disclosure. As shown in FIG14 , the network energy efficiency optimization process includes the following steps S51 to S56 .

[0141] Step S51: detecting the energy consumption of W existing OTN services in the network topology and the creation / deletion of OTN services.

[0142] Step S52: determine whether there is any new OTN service being created or deleted. If M services are being created or K services are being deleted, execute step S53; if no OTN service is being created or deleted, execute step S54.

[0143] Step S53 , periodically taking the (W+MK) OTN services as input to the pre-trained DRL&MCTS model, and inferring to obtain the path distribution with the best global energy efficiency for the (W+MK) OTN services.

[0144] The number of OTN services to be optimized is (W+MK).

[0145] Step S54 , periodically taking W OTN services as input to the pre-trained DRL&MCTS model, and inferring to obtain the path distribution with the best global energy efficiency for the W OTN services.

[0146] The number of OTN services to be optimized is W.

[0147] Step S55: Based on the current path distribution of the OTN service to be optimized and the path distribution with the best global energy efficiency, a progressive path-by-path energy efficiency optimization plan is formulated. The entire optimization process is decomposed into several resource interlocking optimization scenarios and resource independent optimization scenarios, and progressive path-by-path energy efficiency optimization processing is implemented for each of these scenarios in parallel.

[0148] Step S56, determining whether the network operation lifecycle has ended, if not, executing step S51; if it has ended, ending the process.

[0149] The present disclosure relates to the application field of digital twin modeling in the core capability area of ​​"energy efficiency problem identification and optimization" of self-intelligent OTN, and provides a method mechanism for using digital twin technology to realize OTN network energy efficiency problem identification and optimization in a self-intelligent OTN network scenario. The provided network energy efficiency optimization method has little impact on the existing network operation and is not perceived by users.

[0150] An embodiment of the present disclosure also provides a computer device, comprising: at least one processor; a storage device on which at least one computer program is stored; the at least one computer program is executed by the at least one processor, so that the at least one processor implements the network energy efficiency optimization method as described above.

[0151] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon. The computer program is executed by a processor, so that the processor implements the network energy efficiency optimization method as described above.

[0152] It will be appreciated by those skilled in the art that all or some of the steps in the method disclosed above, and the functional modules / units in the device can be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor (such as a central processing unit, a digital signal processor, or a microprocessor), or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0153] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A network energy efficiency optimization method, comprising: Determine the action strategies of multiple services to be optimized according to a pre-trained action strategy learning model; wherein the action strategy learning model is a digital twin deep reinforcement learning network model, and the energy consumption optimization objective function of the action strategy learning model is to minimize the sum of the workload energy consumption and the equipment energy consumption of the multiple services to be optimized; the actual total reward of a trajectory of the action strategy learning model is the timely reward in the establishment state of the last service in the trajectory; According to the action strategy, traverse all service establishment states, determine the service path establishment action with the best global energy efficiency in each service establishment state, and obtain the service path establishment action sequence with the best global energy efficiency; An action sequence is established according to the service path with the best global energy efficiency, and the path distribution with the best global energy efficiency of the multiple services to be optimized is determined.

2. The method of claim 1, wherein: According to the action strategy, traversing all service establishment states, determining the service path establishment action with the best global energy efficiency in the service establishment state includes: For each service establishment state, a Monte Carlo tree search algorithm is used to calculate the action scores corresponding to each alternative service path establishment action in the service establishment state according to the score of executing the alternative service path establishment action, the strategy score of the action strategy in the service establishment state, the number of times the alternative service path establishment action is selected in the service establishment state, and the preset hyperparameters; Determine a target action corresponding to the maximum value of the action score, execute the target action, obtain a next service establishment state, and sequentially generate subsequent service establishment states from the next service establishment state according to the action strategy until the trajectory ends; Calculating a state score of the next business establishment state according to a state value of the next business establishment state and a timely reward obtained from the next business establishment state to the end of the trajectory; According to executing the alternative service path establishment action in the service establishment state, a value of the state score record of the next service establishment state and each state score corresponding to the state score record in the next service establishment state are obtained, an average state score of the next service establishment state is calculated, and the score of executing the alternative service path establishment action is updated by using the average state score; After traversing all service establishment states for a preset number of times, the service path establishment action with the best global energy efficiency in the service establishment state is determined according to the number of times each candidate service path establishment action is selected in the service establishment state.

3. The method of claim 1, further comprising: After establishing an action sequence according to the service path with the best global energy efficiency and determining the distribution of the paths with the best global energy efficiency of the multiple services to be optimized, determining multiple service paths to be optimized according to the distribution of the paths with the best global energy efficiency of the multiple services to be optimized; Each of the service paths to be optimized is optimized one by one.

4. The method of claim 3, wherein: Optimizing each of the service paths to be optimized one by one includes: Dividing the service path to be optimized into optimization scenarios, where the types of the optimization scenarios include resource interlocking optimization scenarios and resource independent optimization scenarios; For each optimization scenario, each service path to be optimized in the optimization scenario is optimized one by one.

5. The method of claim 4, wherein: The type of the optimization scenario is a resource interlocking optimization scenario. For the resource interlocking optimization scenario, optimizing each service path to be optimized in the optimization scenario one by one includes: Determine an unlocking path and a rerouting path of the unlocking path according to the resource interlocking state, wherein the unlocking path is one of the service paths to be optimized in the resource interlocking optimization scenario; Determining a rerouting time of the unlocking path, and migrating the unlocking path to the rerouting path according to the rerouting time to release a network node; Determine the service path to be optimized for this path migration and the optimization time of the service path to be optimized according to the network nodes released by the previous path migration, and migrate the service path to be optimized for this path migration to the corresponding target path according to the optimization time until all the remaining service paths to be optimized in the resource interlocking optimization scenario complete the path migration; An optimization time of the unlock path is determined, and according to the optimization time of the unlock path, the rerouting path is migrated to a corresponding target path.

6. The method of claim 4, wherein: The type of the optimization scenario is a resource-independent optimization scenario. For the resource-independent optimization scenario, optimizing each service path to be optimized in the optimization scenario one by one includes: Determining the optimization time of each of the service paths to be optimized in the resource independent optimization scenario; According to the optimization time, each of the service paths to be optimized in the resource independent optimization scenario is migrated to a corresponding target path one by one.

7. The method according to claim 5 or 6, wherein: Determining the optimization time of the service path to be optimized includes: Predicting the traffic of the service to be optimized; Determining the ebb and flow time of the service to be optimized according to the traffic of the service to be optimized; The optimization time of the business to be optimized is determined according to the tidal fall time; wherein, in the resource interlocking optimization scenario, the optimization time of the business path to be optimized of this path migration is determined according to the tidal fall time of the business path to be optimized of this path migration; in the resource independent optimization scenario, the migration order is determined, and the optimization time of each business to be optimized is determined according to the migration order.

8. The method according to claim 5 or 6, wherein: The time interval between two adjacent optimization times of the service path to be optimized is greater than a preset threshold.

9. A computer device comprising: at least one processor; a storage device having at least one computer program stored thereon; The at least one computer program is executed by the at least one processor, so that the at least one processor implements the network energy efficiency optimization method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor so that the processor implements the network energy efficiency optimization method according to any one of claims 1 to 8.

Images