A deep reinforcement learning based optical transport network routing wavelength optimization method

Abstract

The application discloses a DeepRWA optimization method for optical transport network routing wavelength based on reinforcement learning, which uses a computer as a data processing platform, adopts an SDN framework to flexibly control routing selection and wavelength allocation of the optical transport network, and realizes intelligent processing of RWA based on a deep reinforcement learning strategy; the deep reinforcement learning uses an asynchronous advantage action-comment algorithm A3C, selects a route by considering wavelength usage, and uses an FF algorithm to realize wavelength allocation, so that the routing blocking rate is minimized, and resource utilization efficiency is improved. Simulation experiments are conducted by using the network topology, and results show that, under certain conditions, compared with a traditional KSP-FF algorithm, the performance of the application is significantly improved in terms of the blocking rate and resource utilization rate.

Description

Technical Field

[0001] This invention relates to the field of optical network data processing technology, and in particular to an optimization method for routing wavelength selection in optical networks. Background Technology

[0002] The surge in network services has placed higher demands on backbone network transmission bandwidth. In optical communication networks, selecting appropriate routes and allocating suitable wavelengths to meet the connection needs between source and destination nodes is crucial. Rational wavelength routing optimizes network resource allocation and meets actual service requirements, making it a research hotspot in the field of optical network technology in recent years. How to select appropriate routes and allocate optimized wavelengths for services in a resource-constrained network has a significant impact on improving the utilization of optical network resources, optimizing management, and flexibly controlling the network. Therefore, Routing and Wavelength Assignment (RWA) has become one of the core issues in optical transport networks.

[0003] In the Restricted Flow Assessment (RWA) problem, RWA is generally divided into static RWA and dynamic RWA based on the different services supported by the network. In static RWA, all connection requests between network nodes are predetermined in the form of a service matrix, requiring the establishment of routes and allocation of corresponding wavelengths for these requests. The optimization objective of static RWA is to minimize the number of wavelengths required in the network while maximizing throughput. For static RWA, Zhou Jiaxun proposed a backup wavelength routing and allocation algorithm based on link protection in his paper "Research on Channel Configuration Scheme for Power OTN Networks." This algorithm can save network wavelength resources and improve resource utilization. However, in complex topologies, the complexity and running time of this algorithm increase dramatically. Therefore, this algorithm does not have wide applicability. In dynamic RWA, the number of wavelengths for links in the network is predetermined, and link requests between nodes arrive and leave the network randomly, requiring the allocation of wavelengths for each link request. Research on dynamic RWA is more challenging, and the optimization objective of dynamic RWA is to minimize network congestion rate. With the rapid development of optical networks, dynamic services are predominantly processed in optical networks, leading many scholars to study dynamic RWA. To address the dynamic RWA problem, Wang Botao proposed a multi-objective optimization algorithm based on a hierarchical graph model in his paper "Route Wavelength Allocation Algorithm in WDM Networks." This algorithm can reduce the network congestion rate; however, it drastically increases the complexity of the algorithm in large optical networks. Zhou Shuang et al. (Sun Yi, Zhou Shuang, Lu Jun, Sun Yue, Du Na) proposed a dynamic channel equalization route wavelength allocation algorithm based on the complex power optical transmission network topology in their paper "Dynamic Channel Equalization Route Wavelength Allocation Algorithm for Power Backbone Optical Transmission Networks." This algorithm can achieve differentiated services for different levels of services in complex optical networks and reduce the service congestion rate. However, this algorithm is only applicable to backbone OTNs with wavelength conversion capabilities and cannot be widely used in practical networks. Shen Jianhua et al. designed a route wavelength allocation algorithm based on an ant colony system in their patent "Dynamic Route Wavelength Allocation Method for Optical Networks Based on Ant Colony System." The ant colony algorithm considers the link idle rate index in its transition probability to avoid multiple services selecting the same link, thereby reducing the network congestion rate. However, in actual complex structures with large service volumes, this algorithm has a high time cost and is prone to getting trapped in local optima.

[0004] Due to the dynamic nature of emerging services such as cloud computing and data interconnection, the actual optical network topology is complex, and the requirements for network service carrying capacity are further increasing. Therefore, providing effective routing and wavelength allocation schemes for a large number of bursty services in complex networks is particularly important. The aforementioned routing and wavelength allocation methods are no longer applicable due to their lack of flexibility. It is necessary to rapidly deploy network resources on demand based on service characteristics, and to leverage intelligent algorithms to provide flexible optimization management and control schemes for optical network routing and wavelength selection. Software-defined network (SDN) architecture can provide flexible management and real-time control for a large number of random, bursty, dynamic services, while deep reinforcement learning can provide efficient optimization strategies for the SDN controller. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a DeepRWA (Deep Routing Wavelength Optimization) method for optical transport networks based on reinforcement learning. It employs an SDN (Software-Defined Network) architecture and utilizes machine learning algorithms to obtain efficient routing and wavelength allocation strategies, enabling flexible control of the optical network. By optimizing route selection and wavelength allocation through reinforcement learning, it minimizes routing congestion, exhibits fast convergence speed and good adaptability, and improves resource utilization.

[0006] The technical solutions adopted to overcome the shortcomings of existing technologies are as follows:

[0007] A DeepRWA optimization method for optical transport networks based on reinforcement learning is proposed. It utilizes a computer as a data processing platform, employs an SDN framework to flexibly control routing selection and wavelength allocation in the optical transport network, and achieves intelligent RWA processing based on deep reinforcement learning strategies. Deep reinforcement learning uses the Asynchronous Advantage Action-Comment Algorithm (A3C) and considers wavelength usage when selecting routes. The FF algorithm is used for wavelength allocation to minimize routing congestion and improve resource utilization efficiency. The processing steps of the method are as follows:

[0008] A.DeepRWA is designed based on the SDN architecture, which includes three parts: an optical network module, an SDN controller, and a reinforcement learning module. The optical network module consists of an optical topology, the SDN controller achieves centralized control of the optical network through the OpenFlow protocol, and the reinforcement learning module provides RWA strategies for the SDN controller.

[0009] B. The optical network module first receives the optical path request R from the host. t The server responds to the service, and the SDN controller obtains network topology information and optical path request information. Candidate paths for the service can be obtained through the optical path request.

[0010] C.DeepRWA's reinforcement learning module includes a state, action, and reward system, which reads the state s through a neural network. t The status information includes the source node, destination node, service duration, and available wavelength distribution of the link. The A3C algorithm is used for iterative training, and the output of the neural network is sent to the SDN controller as the RWA strategy.

[0011] D. Based on the actions learned from reinforcement learning, the routing and wavelength of services in the optical network are deployed, and routing and wavelength allocation strategies are provided. With the help of the OpenFlow protocol, the SDN controller distributes the RWA strategy to the optical network module.

[0012] The DeepRWA optimization method for optical transport network routing wavelengths based on reinforcement learning described above, wherein step B involves the following steps: the optical network module is mainly composed of an optical network topology, and the SDN controller obtains node, link, and wavelength information through the optical network topology; in the optical network topology, the source host sends a service request, and the destination host accepts the service request; the service request follows a Poisson distribution, the service time follows a negative exponential distribution, and the arrival rate and average service duration are fixed values; the optical path request includes the source host address, destination host, and service duration information, and the SDN controller obtains the optical path request information through the optical network topology.

[0013] The aforementioned DeepRWA optimization method for optical transport network routing wavelengths based on reinforcement learning, in step C, processes the state information acquired by the SDN controller through a feature processing module, and sends the processed state information to the reinforcement learning module. This state information includes the source node, destination node, service duration, and available wavelength distribution for the link. The output of the neural network is then sent to the SDN controller as the RWA strategy. The state expression is:

[0014] S t ={o,d,τ,{z1,jk,z2,jk}}

[0015] In the formula, o, d, and τ represent the source node, destination node, and service duration of the service request; {z1,jk,z2,jk} represents the distribution of unused wavelengths for a candidate route of the service request, z1,jk represents the number of available wavelengths in the j-th segment of the k-th route, and z2,jk represents the index of the first available wavelength in the j-th segment of the k-th route.

[0016] The DeepRWA optimization method for optical transport network routing wavelengths based on reinforcement learning described above requires setting the routes and wavelengths of services in the optical network in step D. A routing and wavelength allocation scheme represents an action in reinforcement learning, and the action expression is as follows:

[0017] A t =[0,1,2,…,N]

[0018] In the formula, N represents the total number of possible routing and wavelength allocation schemes.

[0019] The aforementioned DeepRWA optimization method for optical transport network routing based on reinforcement learning utilizes the reward system in the reinforcement learning module, which uses historical rewards as feedback and the immediate reward r for generating DeepRWA. t Store the three elements of immediate reward, action, and state data into the experience pool; use the data in the experience pool to train the policy neural network and the value neural network, update the parameters of the neural network, and use the policy neural network to obtain the RWA policy.

[0020] The aforementioned DeepRWA optimization method for optical transport networks based on reinforcement learning uses past rewards as feedback. The optimization objective of the dynamic RWA problem is to minimize the blocking rate, and the value of an action is measured by the success rate of the service. If a service is successfully transmitted through the optical network, the route and wavelength allocation are successful, and at this point, the reward r is immediately given. t =1, otherwise r t =-1, generates an instant reward for DeepRWA.

[0021] The DeepRWA method for optimizing optical transport network routing, based on reinforcement learning, uses data from an experience pool to train and update the neural network's parameters. The neural network iterates continuously to maximize the cumulative reward Γ, as shown in the formula below:

[0022]

[0023] In the formula, t represents the current time, t′ represents a future time, and γ represents the future return discount factor. The RWA strategy is obtained by using a policy neural network, and the RWA strategy is the action with the highest probability in the policy neural network.

[0024] Beneficial effects

[0025] Compared with the prior art, the technological advancements achieved by this invention are as follows:

[0026] This invention addresses the challenges of handling numerous dynamic and bursty services. It employs an SDN network architecture and utilizes machine learning algorithms to acquire efficient routing and wavelength allocation strategies, enabling flexible control of the optical network. Reinforcement learning optimizes routing and wavelength allocation, exhibiting fast convergence and good adaptability. By understanding the state of the optical network and service information, reinforcement learning determines the optimal routing and wavelength allocation strategy and accelerates convergence by setting appropriate rewards. The actions learned through reinforcement learning are then considered as routing and wavelength allocation strategies and distributed to the controller. Considering both congestion rate and resource utilization, a reinforcement learning-based routing and wavelength allocation algorithm, DeepRWA, is proposed. For the routing problem, the A3C algorithm is used to select appropriate routes based on wavelength usage on the links, minimizing the congestion rate. For the wavelength allocation problem, a first-choice algorithm is used to select wavelengths. Through iterative neural network training, the cumulative reward is maximized. After a period of training, the DeepRWA algorithm reduces network congestion rate and improves resource utilization.

[0027] This invention utilizes a typical network topology for simulation experiments. The results show that, under certain conditions, compared with the traditional KSP-FF algorithm, this invention significantly improves performance in terms of blocking rate and resource utilization. Attached Figure Description

[0028] Figure 1 This is a diagram of the overall framework of DeepRWA of the present invention;

[0029] Figure 2 (a) is a schematic diagram of an optical network topology. Figure 2 (b) is a schematic diagram of wavelength usage;

[0030] Figure 3 This is a topology diagram of a 14-node NSFNET;

[0031] Figure 4 This is a comparison chart of the blocking rates of the present invention and the KSP-FF algorithm;

[0032] Figure 5 This is a comparison chart of resource utilization between the present invention and the KSP-FF algorithm.

[0033] The letters in the text represent: RWA (Routing and Wavelength Allocation); o (source node of service request); d (destination node of service request); τ (service duration of service request); s t Status information; z1,jk, the number of available wavelengths for the k-th route and the j-th wavelength segment; z2,jk, the index of the first available wavelength for the j-th wavelength segment of the k-th route; t, the current time; t′, a future time; γ, the future revenue discount factor; Detailed Implementation

[0034] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0035] To address the routing and wavelength allocation problems in optical networks, this invention provides a DeepRWA (Deep Routing Wavelength Optimization) method for optical transport networks based on reinforcement learning, with the framework structure as follows: Figure 1 As shown. The algorithm adopts an SDN architecture, consisting of three parts: an optical network module, a controller, and a reinforcement learning module. The SDN local controller, as the core, first obtains the optical network status and optical path requests based on the data plane, generates RWA policies using a deep reinforcement learning algorithm, and then distributes them to the data plane. The optical network module mainly consists of optical add-drop multiplexers as nodes, interconnected to form a topology. The SDN controller acts as a bridge between the optical network module and the reinforcement learning module, sensing optical network information, wavelength usage, and service request information, providing efficient routing and wavelength allocation strategies for the optical network. The reinforcement learning module uses the information collected by the controller for training and sends the generated actions, including routing paths and wavelength selection, to the controller module. The DeepRWA working process involves intelligently controlling the data plane of the optical transport network through SDN. The specific implementation steps of DeepRWA are as follows:

[0036] A.DeepRWA is designed based on an SDN architecture, consisting of three parts: an optical network module, an SDN controller, and a reinforcement learning module. The optical network module is composed of an optical topology, the SDN controller achieves centralized control of the optical network through the OpenFlow protocol, and the reinforcement learning module provides the SDN controller with efficient RWA strategies.

[0037] B. The optical network module first receives the optical path request R from the host. t The SDN controller obtains network topology information and optical path request information;

[0038] When the optical network is operational, services are initiated by hosts, and servers respond to those services. For a single service request in the topology, a single optical path request from the source node to the destination node is defined as R. t (o,d,τ), where o and d represent the source and destination nodes of the optical path request, and τ represents the duration of the optical path service request.

[0039] This invention employs an RYU controller, which can obtain node information and idle wavelength distribution in the optical network and obtain candidate paths for services through optical path requests.

[0040] C.DeepRWA's reinforcement learning module includes states, actions, and rewards, and reads states s through neural networks. t The A3C algorithm is used for iterative training, and the output of the neural network is sent to the SDN controller as the RWA strategy.

[0041] The core of DeepRWA proposed in this invention is a routing wavelength optimization algorithm based on reinforcement learning. The key to reinforcement learning is the design of states, actions, and rewards. The specific design of the states is as follows:

[0042] For a single service request in the topology, a specific matrix is ​​used to represent the state s. t The wavelength usage of information and candidate paths, and the state settings of this invention are as follows: Figure 2 As shown, where Figure 2 (a) is an optical network topology; Figure 2 (b) is a schematic diagram of wavelength usage in an optical network link, defined as a matrix of size 1×(2|V|+1+2JK), as shown in equation (1):

[0043] S t ={o,d,τ,{z1,jk,z2,jk}} (1)

[0044] In the formula, o, d, and τ represent the source node, destination node, and service duration of the service request; k∈[1,2,…,K], j∈[1,2,…,J], and K is the number of all candidate routes for the service request (e.g., ...). Figure 2 (a) Dashed line portion), J represents all available wavelength segments for a certain route. A single available wavelength segment contains at least one available wavelength, and all wavelengths within the segment are in consecutive positions, such as... Figure 2 In (b), the white portion, V, represents the number of nodes in the topology. Different numbers of candidate routes lead to different states, therefore a fixed number of candidate routes must be used. Reinforcement learning can gain greater benefits by exploring more candidate routes; however, the convergence time will also be longer. Too few candidate routes result in smaller reinforcement learning benefits; too many candidate routes slow down the algorithm's convergence speed. This invention sets K=5. {z1,jk,z2,jk} represents the distribution of unused wavelengths for a candidate route requested by the service. z1,jk represents the number of available wavelengths in the j-th segment of the k-th route, and z2,jk represents the index of the first available wavelength in the j-th segment of the k-th route. See the specific example below. Figure 2 In (a), assume that the service has two routes to choose from: 1-2-4 is route 1 (k=1) and 1-3-5-4 is route 2 (k=2). Figure 2In (b), the wavelength usage of routes 1 and 2 is divided into two parts: For the upper route 1, the available wavelengths in the first segment are z1,1 + 1 = 3, and the starting index of the available wavelength set for the first segment of route 1 is z2,1 + 1 = 0; for the lower route 2, the available wavelengths in the first segment are z1,1 + 2 = 3, and the starting index of the available wavelength set for the second segment of route 1 is z2,1 + 2 = 9. For the lower route 2, the available wavelengths in the first segment are z1,1 + 2 = 4, and the starting index of the available wavelength set for the first segment of route 2 is z2,1 + 2 = 2; for the lower route 2, the available wavelengths in the second segment are z1,2 + 2 = 4, and the starting index of the available wavelength set for the second segment of route 2 is z2,2 + 2 = 8. Therefore... Figure 2 The state is represented as: S t ={o,d,τ,z1,1 1~z2,2 1,z1,1 2~z2,2 2}.

[0045] D. Using the OpenFlow protocol, the SDN controller distributes the RWA policy to the optical network module;

[0046] The DeepRWA algorithm deploys routes and wavelengths for services in an optical network based on reinforcement learning actions. A routing and wavelength allocation scheme represents one reinforcement learning action. For a service request, the DeepRWA algorithm selects a routing path from K candidate paths in the optical network topology, calculates the number of available wavelengths I based on the state, and selects an available wavelength from the I available wavelengths to transmit data. Therefore, the action space contains K*I actions. The action expression is:

[0047] A t =[0,1,2,…,N]

[0048] In the formula, N represents the total number of possible routing and wavelength allocation schemes.

[0049] E. Updates to the reward system and experience pool

[0050] The reward system will use past rewards as feedback and generate immediate rewards for DeepRWA. t The reward setting significantly impacts the algorithm's convergence speed and action selection. Since the optimization objective of the dynamic RWA problem is to minimize the blocking rate, the success rate of the service is used to measure the value of the action. The immediate reward serves as a marker, recording whether the current service request was successful. If the DeepRWA algorithm successfully enables a service to be transmitted through the optical network (i.e., routing and wavelength allocation are successful), the immediate reward r... t =1, otherwise r t =-1. In this invention, the neural network iterates continuously to maximize the cumulative return Γ, and the cumulative return formula is shown in (2):

[0051]

[0052] In the formula, t represents the current time, t′ represents a future time, and γ represents the future return discount factor.

[0053] Store the three elements of immediate reward, action, and state data into the experience pool; use the data in the experience pool to train the policy neural network and the value neural network, update the parameters of the neural network, and use the policy neural network to obtain the RWA policy.

[0054] A trained algorithm model was selected and simulated using a 14-node NSFNET topology. This network has 14 nodes and 44 links. It is assumed that dynamic service requests follow a Poisson distribution, service time follows a negative exponential distribution, arrival rate and average service time are 1 / 12 and 1 / 3 respectively, and each link contains 18 wavelengths. Routing and wavelength allocation strategies are obtained using the DeepRWA algorithm. The blocking rate is calculated by the ratio of blocked services to total services, and the resource utilization rate is calculated by the ratio of used wavelengths to total wavelengths. Simulation results are shown below. Figure 4 and Figure 5 The results show that, compared with the KSP-FF algorithm, the blocking rate of the present invention is reduced by 0.06, and the resource utilization is increased by 0.02.

Claims

1. A DeepRWA optimization method for optical transport networks based on reinforcement learning. It utilizes a computer as a data processing platform, employs an SDN framework to flexibly control routing selection and wavelength allocation in the optical transport network, and achieves intelligent RWA processing based on deep reinforcement learning strategies. Deep reinforcement learning uses the Asynchronous Advantage Action-Comment Algorithm (A3C) and considers wavelength usage for route selection. The FF algorithm is used for wavelength allocation, minimizing routing congestion and improving resource utilization efficiency. Its key features are... The processing steps of the method are as follows: A. DeepRWA is designed based on the SDN architecture, which includes three parts: an optical network module, an SDN controller, and a reinforcement learning module. The optical network module consists of an optical topology, the SDN controller achieves centralized control of the optical network through the OpenFlow protocol, and the reinforcement learning module provides RWA strategies for the SDN controller. B. The optical network module first receives the optical path request from the host. R t The server responds to the service, and the SDN controller obtains network topology information and optical path request information. Candidate paths for the service can be obtained through the optical path request. C. DeepRWA's reinforcement learning module includes state, action, and reward, and reads the state through a neural network. s t The status information includes the source node, destination node, service duration, and available wavelength distribution of the link. The A3C algorithm is used for iterative training, and the output of the neural network is sent to the SDN controller as the RWA strategy. D. Based on the actions learned from reinforcement learning, the routing and wavelength of services in the optical network are configured, and routing and wavelength allocation strategies are provided. With the help of the OpenFlow protocol, the SDN controller distributes the RWA strategy to the optical network module. Step B specifically involves the following: The optical network module is mainly composed of an optical network topology. The SDN controller obtains node, link, and wavelength information through the optical network topology. In the optical network topology, the source host sends a service request, and the destination host accepts the service request. The service request follows a Poisson distribution, the service time follows a negative exponential distribution, and the arrival rate and average service duration are fixed values. Optical path request includes source host address, destination host, and service duration information. The SDN controller obtains the optical path request information through the optical network topology. In step C, the state information acquired by the SDN controller is processed by the feature processing module, and the processed state information is sent to the reinforcement learning module. This state information includes the source node, destination node, service duration, and available wavelength distribution of the links. The output of the neural network is sent to the SDN controller as the RWA (Reinforcement Learning and WA) policy. The state expression is: S t ={ o , d , τ ,{z1, jk ,z2, jk }} In the formula, o , d and τ Indicates the source node, destination node, and service duration of the business request; { z1,jk , z2,jk The} symbol represents the distribution of unused wavelengths for a candidate route in a business request. z1,jk Indicates the first k Route number j The number of wavelengths available in a segment of wavelength, z2,jk Indicates the first k Route number j The index of the first available wavelength in the wavelength segment.

2. The DeepRWA optimization method for optical transport network routing wavelengths based on reinforcement learning according to claim 1, characterized in that, Step D requires configuring the routing and wavelengths of services in the optical network. A routing and wavelength allocation scheme represents an action in reinforcement learning, and the action expression is as follows: A t =[0,1,2,…,N] In the formula, N represents the total number of possible routing and wavelength allocation schemes.

3. The DeepRWA optimization method for optical transport network routing wavelengths based on reinforcement learning according to claim 1, characterized in that, The reward system in the reinforcement learning module uses past historical rewards as feedback and generates immediate rewards for DeepRWA. t Store the three elements of immediate reward, action, and state data into the experience pool; use the data in the experience pool to train the policy neural network and the value neural network, update the parameters of the neural network, and use the policy neural network to obtain the RWA policy.

4. The DeepRWA optimization method for optical transport network routing wavelengths based on reinforcement learning according to claim 1, characterized in that, Historical rewards are used as feedback; the optimization objective of the dynamic RWA problem is to minimize the blocking rate, and the value of actions is measured by the success rate of services; if a service is successfully transmitted through the optical network, the routing and wavelength allocation are successful, and an immediate reward r is given. t =1, otherwise r t =-1, generates instant rewards for DeepRWA.

5. The DeepRWA optimization method for optical transport network routing wavelengths based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning module uses data from the experience pool to train the neural network and update its parameters; the neural network iterates continuously to accumulate gains. G The formula for maximizing cumulative returns is shown below: In the formula, t represents the current time, and t′ represents a future time. γ This represents the future return discount factor; the RWA strategy is obtained using a policy neural network, where the RWA strategy is the action with the highest probability selected in the policy neural network.

Images