Cellular optical network optical switching routing method based on improved PPO algorithm

Abstract

The invention discloses a honeycomb type optical network optical switching routing method based on an improved PPO algorithm, and the method comprises the steps: building a honeycomb type optical network simulation array, carrying out the predefinition of an optical network according to the topological structure characteristics of the honeycomb type optical network simulation array, building an optical switching routing model based on deep reinforcement learning according to an optical network model, and training the optical switching routing model through an improved PPO algorithm, and obtaining an optical routing path with the minimum loss by using the trained optical switching routing model. According to the method, the defects that in the prior art, the algorithm feature learning capacity is limited, the expression capacity for complex functions is weak, and optimization target parameters are single are effectively overcome, and therefore the network performance, reliability and the utilization rate are improved.

Description

Technical Field

[0001] The invention relates to the technical field of optical switching, and in particular to an optical switching routing method of a cellular optical network based on an improved PPO algorithm. Background Art

[0002] With the widespread popularity of 5G networks, the data traffic business in the information society has developed rapidly, and the data traffic has exploded, which has put forward higher requirements for data information transmission in data centers. In order to meet this demand, silicon-based optical interconnection technology with advantages such as high bandwidth, low latency and low energy consumption has gradually replaced electrical interconnection technology and become an effective solution for short-distance interconnection in integrated circuit chips. As the core device in silicon-based optoelectronic integrated chips, optical switches can be cascaded into optical networks for high-speed data exchange. Optical switch networks can be divided into forward transmission networks and multi-directional transmission networks. Cellular optical networks belong to multi-directional transmission networks. Compared with forward transmission networks, they have higher transmission freedom, and compared with multi-directional transmission networks such as triangles and quadrilaterals, they have excellent performance in reconfigurability, crosstalk and area utilization. However, many studies have shown that power loss and crosstalk noise in silicon-based electro-optical switch networks will affect the transmission of optical signals. High loss and high crosstalk may even lead to communication being blocked, limiting the expansion of network scale.

[0003] By using routing optimization algorithms to select routing paths with low power consumption and crosstalk, the transmission performance of optical signals in optical switch networks can be effectively improved. Currently, the routing algorithm widely used in optical switch networks is the dimension-order routing algorithm, but the dimension-order routing algorithm cannot reasonably select optical routing paths with low power consumption and crosstalk according to the actual network conditions, and it is difficult to meet the routing requirements of large-scale or complex topology optical networks. Therefore, a lot of research is needed on routing algorithms for large-scale multi-directional controllable electro-optical switch networks.

[0004] With the advent of the era of artificial intelligence and the breakthroughs made in the field of mathematical foundations, as one of the core technologies in the field of artificial intelligence, deep reinforcement learning (DRL) has been increasingly used to solve practical problems. Among them, the PPO algorithm has the advantages of stable training and high robustness, and can be well applied to optical network routing problems. However, the traditional PPO algorithm has problems such as slow convergence speed and easy deadlock. In order to solve the above problems in the existing technical algorithms, promote and enrich the research on algorithms related to dynamic routing and optical signal transmission of optical switch networks, the research on optical switching routing methods for cellular networks using the improved PPO algorithm will become a beneficial exploration in the field of optical switching technology research. Summary of the invention

[0005] The purpose of the present invention is to provide a cellular optical network optical switching routing method based on an improved proximal policy optimization (PPO) algorithm, to build a cellular optical network simulation array, to predefine the optical network according to its topological structure characteristics, and to construct an optical switching routing model based on deep reinforcement learning according to the optical network model, to train the optical switching routing model through the improved PPO algorithm, to obtain the optical routing path with the minimum loss using the trained optical switching routing model, and to improve the network performance, reliability and utilization.

[0006] To achieve the above object, the present invention provides a cellular optical network optical switching routing method based on an improved PPO algorithm, comprising the following steps:

[0007] S1: Building an N×M cellular optical network simulation array, and predefining an optical network according to the topological structure characteristics of the cellular network simulation array;

[0008] S2: Based on the optical network model in step S1, an optical switching routing model based on deep reinforcement learning is built, the optical switching routing problem is described as a Markov decision process, and the state space, action space, actor network actor, critic network critic and reward function R of the improved PPO algorithm are established;

[0009] S3: receiving a request to establish a route, and training the optical switching routing model based on step S2 according to the improved PPO algorithm;

[0010] S4: Using the optical switching routing model trained in step S3, obtain the optical routing path with the minimum loss.

[0011] Furthermore, in the optical network simulation array of step S1, each optical network node is composed of a basic optical switch unit, each basic optical switch unit is provided with 6 ports, 3 of which are input ports and 3 are output ports, and optical signals enter and exit the optical switch from ports in three directions, each direction has an input port and an output port, light is input from the input port in one direction, and is output from the output ports in the other two directions; each basic optical switch unit is connected to other optical switch units through a bidirectional optical waveguide link, except that the outer optical switch unit is connected to two optical switch units, and the remaining optical switch units are connected to three optical switch units; a simple weighted directed graph G(V,E,L) is used to represent the cellular optical network simulation array, wherein V={v1,v2,…,v n} is the set of basic optical switch units in the network; E = {e ij} indicates that from unit v i to v j The optical waveguide link set; L = {l ij} indicates that from unit v i to v j Optical waveguide link ij The transmission loss set.

[0012] Furthermore, in step S2, both the actor network and the critic network adopt a long short-term memory network (LSTM), and the node state information collected by interacting with the environment, i.e., the state space, is input into the LSTM neural network as a feature vector, and the memory unit in the LSTM neural network is used to select the information to be remembered and forgotten in the sample. Both the actor network and the critic network adopt a tanh activation function, wherein the actor network is divided into an encoder and a decoder.

[0013] Furthermore, in step S3, the optical switching routing model is trained according to the PPO algorithm, specifically:

[0014] S3.1: Initialize the environment and agent state s t =s0, initialize the starting point x0 and end point d of the optical switch unit, initialize the experience pool M, and initialize the maximum number of iterations K iteration The maximum number of actions K of the agent in each iteration step , let the current iteration number k iteration = 0, the current agent action times k step =0;

[0015] S3.2: In state s t Next, the probabilities of all actions are obtained through the actor network, and action a is selected based on probability sampling. t , and obtain the corresponding reward r t+1 and enter state s t+1 , where s t Indicates that the agent is in the kth step The state after the action; a t Indicates that the agent is in state s t Enter t+1 The action performed; t Represents the agent from state s t Execute action a t Enter t+1 Rewards received;

[0016] S3.3: Calculate the state value v(s) through the critic network t ), get an experience: (s t ,a t ,r t+1 ,v(s t ,logP(a t,s t )), where logP(a t ,s t ) is the importance weight, and experience is put into the experience pool M;

[0017] S3.4: Determine whether to end, that is, state s t+1 Is it the end point d or the maximum number of actions reached? If s t+1 ≠d and k step ≠K step , let k step =k step +1, return to step S3.2; otherwise, go to step S3.5;

[0018] S3.5: Update the actor network and critic network;

[0019] S3.6: When the agent reaches the maximum number of iterations, k iteration =K iteration When the model training is completed, the starting point x0 and the end point d of the optical switch unit are input, and the trained model is run to output the routing result. Otherwise, the environment and the agent state s are initialized. t =s0, let k iteration =0, k step =0, go to step S3.2.

[0020] Furthermore, the formula for obtaining the probability of all actions through the actor network in step S3.2 is as follows:

[0021]

[0022] Among them, v T represents the coefficient matrix of the attention mechanism; W1 and W2 represent the coefficient parameters obtained through training; e j represents the hidden layer output of the encoder at step j; d i represents the hidden layer output of the decoder at step i; It is a function that calculates the relevance of each action to the current state; softmax represents a definite variable whose output value is between (0,1).

[0023] Furthermore, in step S3.5, the actor network is updated, specifically:

[0024] The advantage function is calculated according to formula (3), and based on the advantage function, the actor network is updated according to formula (4) to maximize the objective function.

[0025]

[0026] Furthermore, in step S3.5, the critic network is updated, specifically:

[0027] G t =r t+1 +γr t+2 +…+γ T-t r T+1 +γ T+1-t v(s T+1 ),

[0028] Where v(s T+1 ) is the estimated value of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the cellular optical network optical switching routing method based on the improved PPO algorithm of the present invention;

[0030] Figure 2 A topological diagram of an N×M cellular optical network provided in an embodiment;

[0031] Figure 3 A schematic diagram of an optical switching routing model based on deep reinforcement learning in the present invention;

[0032] Figure 4 The figure is a flow chart of training an optical switching routing model according to the PPO algorithm in the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] This embodiment provides a cellular optical network optical switching routing method based on an improved PPO algorithm. Figure 1 , the optical switching routing method comprises:

[0035] S1: Building an N×M cellular optical network simulation array, and predefining an optical network according to the topological structure characteristics of the cellular network simulation array;

[0036] In this embodiment, the N×M cellular optical network is as follows: Figure 2As shown, the network consists of N rows and M columns of hexagonal grids. Each hexagonal grid contains 6 optical network nodes and 6 optical links. Each optical network node consists of a basic optical switch unit. Each basic optical switch unit has 6 ports, of which 3 ports are input ports and 3 ports are output ports. Optical signals enter and exit the optical switch from ports in three directions. Each direction has an input port and an output port. Light is input from the input port in one direction and is output from the output ports in the other two directions. Each basic optical switch unit is connected to other optical switch units through a bidirectional optical waveguide link. Except for the outer optical switch unit connected to two optical switch units, the remaining optical switch units are connected to three optical switch units. A simple weighted directed graph G(V,E,L) is used to represent the cellular optical network simulation array, where V = {v1,v2,…,v n} is the set of basic optical switch units in the network; E = {e ij} indicates that from unit v i to v j The optical waveguide link set; L = {l ij} indicates that from unit v i to v j Optical waveguide link ij For the cellular optical network, the optical signal is input into the optical network through the outer basic optical switch unit, routed in the optical network, and finally output from the optical network through the outer optical switch unit. In this process, the optical switch unit sequence {v i ,…,v j ,…,v k} is a light routing path, where v i ,v j ,v k ∈V.

[0037] S2: Based on the optical network model in step S1, an optical switching routing model based on deep reinforcement learning is built, the optical switching routing problem is described as a Markov decision process, and the state space, action space, reward function, actor network actor and critic network critic of the PPO algorithm are established;

[0038] In this embodiment, the agent represents an optical signal, and the state space includes the distance between the optical switch unit where the agent is currently located and the destination optical switch unit, and the adjacency matrix of the optical network composed of all optical switch units. The action space includes six different directions that the agent can route in the optical switch unit.

[0039] In order to avoid sparse rewards, the action decisions of the agent in different states are given reward values, which are mainly divided into two types: arrival reward and route reward.

[0040] Arrival reward: The last optical switch unit that the agent passes through when exiting the optical network is set as the end point. Once the agent reaches the end point, it will receive an arrival reward.

[0041] Routing reward: The routing reward is negatively correlated with the transmission loss caused by the current agent action and takes a negative value. The greater the transmission loss, the smaller the routing reward.

[0042] The total reward of the agent is the weighted sum of the above two rewards, and the weights can be determined and adjusted based on the actions.

[0043] Optical switching routing model based on deep reinforcement learning Figure 3 As shown in the figure, both the actor network and the critic network use LSTM. The node state information collected by the interaction with the environment, i.e., the state space, is input into the LSTM neural network as a feature vector. The memory unit in the LSTM neural network is used to select the information that needs to be remembered and forgotten in the sample to speed up the convergence of the algorithm. Both the actor network and the critic use the tanh activation function. The actor network is divided into an encoder and a decoder. The encoder converts the state space X = {X0, X1, …, X n} is converted into an intermediate vector expression as the original input sequence and input into the decoder. In each step, the decoder uses the attention mechanism and the softmax function to calculate the probability of each element in the input sequence, that is, the action probability distribution of the optical switch unit of the next hop selection route. The critic network selects an action based on the probability distribution obtained by the actor and calculates the state value obtained after executing the action, that is, the transmission loss of the optical waveguide link from the current optical switch unit to the selected next hop optical switch unit.

[0044] S3: receiving a request to establish a route, and training the optical switching routing model based on step S2 according to the PPO algorithm;

[0045] In this embodiment, if Figure 4 The optical switching routing model is trained according to the PPO algorithm, specifically:

[0046] S3.1: Initialize the environment and agent state s0, initialize the starting point x0 and end point d of the optical switch unit, initialize the experience pool M, and initialize the maximum number of iterations K iteration The maximum number of actions K of the agent in each iteration step , let the current iteration number k iteration = 0, the current agent action times k ste p = 0;

[0047] S3.2: In state s t Next, the probabilities of all actions are obtained through the actor network, and action a is selected based on probability sampling. t, and obtain the corresponding reward r t+1 and enter state s t+1 , where s t Indicates that the agent is in the kth step The state after the action; a t Indicates that the agent is in state s t Enter t+1 The action performed; t Represents the agent from state s t Execute action a t Enter t+1 Rewards received;

[0048] In the embodiment, the formula for obtaining the probability of all actions through the actor network is as follows:

[0049]

[0050] Among them, v T represents the coefficient matrix of the attention mechanism; W1 and W2 represent the coefficient parameters obtained through training; e j represents the hidden layer output of the encoder at step j; d i represents the hidden layer output of the decoder at step i; It is a function that calculates the relevance of each action to the current state; softmax represents a definite variable whose output value is between (0,1).

[0051] S3.3: Calculate the state value v(s) through the critic network t ), get an experience: (s t ,a t ,r t+1 ,v(s t ,logP(a t ,s t )), where logP(a t ,s t ) is the importance weight, and experience is put into the experience pool M;

[0052] S3.4: Determine whether to end, that is, state s t+1 Is it the end point d or the maximum number of actions reached? If s t+1 ≠d and k step ≠K step , let k step =k step +1, return to step S3.2; otherwise, go to step S3.5;

[0053] S3.5: Update the actor network and critic network;

[0054] In the embodiment, the calculation formula for updating the actor network is as follows:

[0055]

[0056] where δ t =r t +γV(s t+1 )-V(s t ), the advantage function can be calculated according to formula (3), and based on the advantage function, the actor network is updated by maximizing the objective function according to formula (4). reflects the degree of difference between the new and old strategies. This indicates that the current strategy is better and it is necessary to increase the strategy update amplitude. However, if the update amplitude is too large, the difference between the new and old strategies will be too large, i.e., r t >1+∈, it can be seen from formula (4) that the clip operation and min operation will be restricted at this time, preventing over-optimization and ensuring the stability of the algorithm.

[0057] In the embodiment, after the Actor network is updated, the data extracted from the experience pool is used to update the critic network. The calculated discounted return is subtracted from the value predicted by the critic network for the current state, and the network is trained using MSEloss as the loss function. The calculation formula for updating the critic network is as follows:

[0058] G t =r t+1 +γr t+2 +…+γ T-t r T+1 +γ T+1-t v(s T+1 )

[0059] Where v(s T+1 ) is the estimated value of the network.

[0060] S3.6: When the agent reaches the maximum number of iterations, k iteration =K iteration When the model training is completed, the starting point x0 and the end point d of the optical switch unit are input, and the trained model is run to output the routing result. Otherwise, the environment and the agent state s are initialized. t =s0, let k iteration =0, k step =0, go to step S3.2.

[0061] S4: Use the optical switching routing model trained in step S3 to obtain the optical routing path with the minimum loss. In the embodiment, whenever an optical signal is input into the optical switch unit, the starting point x0 and the end point d are input into the optical switching routing model, and then the model outputs the routing decision information of the current state s, we can get the next state s', and then input the state s' into the model, and get the next state again, repeat the above operation, and finally reach the end point d. Therefore, a path from the starting point x0 to the end point d can be obtained, and this path is the optical routing path with the minimum loss.

Claims

1. The object of the present invention is to provide a cellular optical network optical switching routing method based on an improved PPO algorithm, characterized in that: include: S1: Building an N×M cellular optical network simulation array, and predefining an optical network according to the topological structure characteristics of the cellular network simulation array; S2: Based on the optical network model in step S1, an optical switching routing model based on deep reinforcement learning is built, the optical switching routing problem is described as a Markov decision process, and the state space, action space, actor network actor, critic network critic and reward function R of the PPO algorithm are established; S3: receiving a request to establish a route, and training the optical switching routing model based on step S2 according to the PPO algorithm; S4: Using the optical switching routing model trained in step S3, obtain the optical routing path with the minimum loss.

2. The cellular optical network optical switching routing method based on the improved PPO algorithm according to claim 1 is characterized in that: In step S1, each optical network node is composed of a basic optical switch unit, each basic optical switch unit is provided with 6 ports, of which 3 ports are input ports and 3 ports are output ports. Optical signals enter and exit the optical switch from ports in three directions, each direction has an input port and an output port, light is input from the input port in one direction, and is output from the output ports in the other two directions; each basic optical switch unit is connected to other optical switch units through a bidirectional optical waveguide link, except that the outer layer optical switch unit is connected to two optical switch units, and the remaining optical switch units are connected to three optical switch units; a simple weighted directed graph G(V,E,L) is used to represent the cellular optical network simulation array, where V = {v1,v2,…,v n } is the set of basic optical switch units in the network; E = {e ij } indicates that from unit v i to v j The optical waveguide link set; L = {l ij } indicates that from unit v i to v j Optical waveguide link ij The transmission loss set.

3. The cellular optical network optical switching routing method based on the improved PPO algorithm according to claim 1 is characterized in that: In step S2, both the actor network and the critic network use LSTM, and the node state information collected by interaction with the environment, namely the state space, is input into the LSTM neural network as a feature vector. The memory unit in the LSTM neural network is used to select the information to be remembered and forgotten in the sample. Both the actor network and the critic network use the tanh activation function; wherein, the actor network is divided into an encoder and a decoder.

4. The cellular optical network optical switching routing method based on the improved PPO algorithm according to claim 1 is characterized in that: In step S3, the optical switching routing model is trained according to the improved PPO algorithm, including: S3.1: Initialize the environment and agent state s0, initialize the starting point x0 and end point d of the optical switch unit, initialize the experience pool M, and initialize the maximum number of iterations K iteration The maximum number of actions K of the agent in each iteration step , let the current iteration number k iteration = 0, the current agent action times k step =0; S3.2: In state s t Next, the probabilities of all actions are obtained through the actor network, and action a is selected based on probability sampling. t , and obtain the corresponding reward r t+1 and enters state s t+1 , where s t Indicates that the agent is in the kth step The state after the action, a t Indicates that the agent is in state s t Enter t+1 The action performed, r t Represents the agent from state s t Execute action a t Enter t+1 Rewards received; S3.3: Calculate the state value v(s) through the critic network t ), get an experience: (s t ,a t ,r t+1 ,v(s t ,logP(a t ,s t )), where logP(a t ,s t ) is the importance weight, and experience is put into the experience pool M; S3.4: Determine whether to end, that is, state s t+1 Is it the end point d or the maximum number of actions reached? If s t+1 ≠d and k step ≠K step , let k step =k step +1, return to step S3.2; otherwise, go to step S3.5; S3.5: Update the actor network and critic network; S3.6: When the agent reaches the maximum number of iterations, k iteration =K iteration When the model training is completed, the starting point x0 and the end point d of the optical switch unit are input, and the trained model is run to output the routing result. Otherwise, the environment and the agent state s are initialized. t =s0, let k iteration =0, k step =0, go to step S3.

2.

5. According to claim 4, the optical switching routing model is trained according to the improved PPO algorithm, characterized in that: In step S3.2, the probability calculation formula for all actions obtained through the actor network is as follows: Among them, v T represents the coefficient matrix of the attention mechanism; W1 and W2 represent the coefficient parameters obtained through training; e j represents the hidden layer output of the encoder at step j; d i represents the hidden layer output of the decoder at step i; It is a function that calculates the relevance of each action to the current state; softmax represents a definite variable whose output value is between (0,1).

6. The method of training an optical switching routing model according to the improved PPO algorithm according to claim 4, characterized in that: In step S3.5, the formula for updating the actor network is as follows: among them t =r t +γV(s t+1 )-V(s t ), 7. The method of training an optical switching routing model according to the improved PPO algorithm according to claim 4, characterized in that: In step S3.5, the formula for updating the critic network is as follows: G t =r t+1 +γr t+2 +…+γ T-t r T+1 +γ T+1-t v(s T+1 ) Where v(s T+1 ) is the estimated value of the network.

Images