Photon network-on-chip routing method based on PPO-GRU algorithm

Abstract

The invention discloses a photon network-on-chip routing method based on a PPO-GRU algorithm, and the method comprises the steps: firstly constructing a PNoC simulation array, and building a PNoC routing model based on deep reinforcement learning in combination with optical signal transmission characteristics; according to the model core, on the basis of a PPO algorithm, an invalid action shielding mechanism and a GRU module are fused, and the path searching efficiency is effectively improved. And meanwhile, by adopting dominant function normalization and reward scaling technologies, the stability and convergence reliability of model training are remarkably enhanced. By using the trained routing model, the low-loss optical routing path can be intelligently selected, so that the network performance, the reliability and the resource utilization rate of the PNoC are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of photon on-chip networks, and in particular to a photon on-chip network routing method based on a PPO-GRU algorithm. Background Art

[0002] With the rapid development of data traffic in the information society and the exponential growth of artificial intelligence models, how to process and transmit data at faster rates and lower power consumption has become a core issue that needs to be addressed. Optical interconnect technology, with its significant advantages such as high bandwidth, low transmission latency, low power consumption, and good compatibility with complementary metal semiconductor processes, offers a promising solution for achieving high-speed, energy-efficient data transmission on silicon-based optoelectronic integrated circuits, becoming a key path to breaking through the computing bottleneck of the "post-Moore era." However, the large-scale application of optical interconnect technology in photonic networks-on-chip (PNoC) for medium- and short-range interconnection scenarios still faces many technical challenges. A particularly prominent issue is the transmission loss of optical signals during waveguide transmission and device coupling. These losses require higher laser drive power to maintain signal integrity, significantly increasing the overall system power consumption and severely restricting the scalability of PNoCs.

[0003] Optimizing routing algorithms and reducing transmission loss are crucial for improving network performance. Currently, PNoCs can be categorized as forward-only networks and recursive networks. Compared to forward-only networks, recursive networks offer superior path diversity, crosstalk, and area utilization. However, most current PNoC algorithms focus primarily on forward-only networks, making them inadequate for the routing requirements of large-scale recursive PNoCs.

[0004] Deep reinforcement learning (DRL), due to its ability to make autonomous decisions in complex environments, has become an important tool for optimizing dynamic routing strategies. However, current research on PNOC routing algorithms based on DRL methods is limited, and they suffer from issues such as low sample efficiency, poor robustness, and unstable training. These issues become increasingly prominent as the scale of PNOC networks increases. To address these issues in existing algorithms, promote and enrich research on PNOC dynamic routing and optical signal transmission algorithms, and effectively address the exploration efficiency, reliability, and stability issues of PNOC routing, research on PNOC routing algorithms using DRL methods will be a valuable exploration in the PNOC research field. Summary of the Invention

[0005] The present invention aims to propose a PNoC routing method based on Proximal Policy Optimization (PPO) and Gate Recurrent Unit (GRU). A PNoC simulation array is constructed, and a DRL-based PNoC routing model is built based on the optical signal transmission characteristics and the PNoC simulation array. Based on the PPO algorithm, invalid action shielding and the GRU module are combined to improve path search efficiency. Advantage function normalization and reward scaling are used to enhance the stability and convergence reliability of model training. The trained routing model is used to obtain lossy optical routing paths, thereby improving network performance, reliability, and utilization.

[0006] To achieve the above objectives, the present invention provides a photonic on-chip network routing method based on the PPO-GRU algorithm, comprising the following steps:

[0007] S1: Build a PNoC simulation array, obtain the transmission loss of optical signals under different switching states, and predefine an optical network model based on the topological structure characteristics of the PNoC simulation array;

[0008] S2: Based on the PNoC model in step S1 and the optical signal transmission characteristics, a DRL-based PNoC routing model is built. The routing problem is described as a Markov decision process, and the intelligent agent, state space S, action space A, reward function R, and experience replay pool in reinforcement learning are established.

[0009] S3: Train the PNoC routing model based on step S2 according to the PPO-GRU algorithm;

[0010] S4: Using the PNoC routing model trained in step S3, obtain the optical routing path with the minimum loss.

[0011] Furthermore, the PNoC simulation array of step S1 is composed of basic optical switch units and optical waveguide links, and is represented by a simple weighted directed graph. To represent the PNoC simulation array, where the node set Represents the set of all basic optical switch units in the network, the edge set Represents the set of all optical waveguide links, loss set Indicates that the optical signal comes from The input is from port p and the output is from port q. The transmission loss set is: each basic optical switch unit has m ports, where m ≥ 2. The optical signal can enter the optical switch from any port and exit the optical switch from the remaining m-1 ports. Each basic optical switch unit is connected to other optical switches through an optical waveguide link. A two-dimensional coordinate system is used to represent the position of the optical switch in the network, and the position of the optical switch unit is defined as (x, y).

[0012] Furthermore, in step S2, the intelligent agent includes an actor and a critic, wherein the actor is responsible for learning the routing strategy and the critic is responsible for commenting on the quality of the strategy. Both use GRU as the neural network structure, use orthogonal initialization of GRU network parameters, and use tanh activation function.

[0013] Furthermore, in step S2, the state space includes three parts: the moving direction of the agent, the relative coordinates and the feasibility of movement; the moving direction of the agent The direction in which the agent moves from the previous node to the current node; relative coordinates Refers to the axial coordinate difference between the current node and the destination node of the intelligent agent Mobile feasibility Refers to whether it is feasible for the agent to move from the current node to the adjacent node, using 0 / 1 to indicate infeasible / feasible; a one-dimensional tensor is used to encode the state of each time step and define the state space .

[0014] Furthermore, in step S2, the action space is the set of all possible transformations of the agent's movement direction.

[0015] Furthermore, in step S3, the reward function contains two parts, one of which is the negative value of the transmission loss obtained by the agent after each action, and the other is the arrival reward obtained by the agent when it reaches the destination node. When , the reward obtained by the agent is:

[0016] (1)

[0017] in, is a non-zero scaling factor, , or 1, is a positive value associated with the size of the network.

[0018] Furthermore, in step S3, the PNoC routing model is trained according to the PPO-GRU algorithm, specifically:

[0019] S3.1: Set the maximum number of iterations , GRU update frequency The maximum number of actions per round , let the current iteration number ;

[0020] S3.2: Randomly initialize the starting point , current node and end point ,in , initialize the environment and get the initial state , initialize the experience replay pool, and make the number of actions of the current round agent ;

[0021] S3.3: Current state In the GRU of the input actor, the unnormalized action probability prediction value output by the GRU is masked for invalid actions to obtain the action probability distribution , Actor selects actions based on probability sampling , environment according to renew And output The rewards , the state at the next moment and whether the destination has been reached ;

[0022] S3.4: Use reward scaling Processing, dynamically calculate the standard deviation of the discount of the total reward in a round , then Divide by the standard deviation to get ;Will and Critic The value function predicts the value Stored as a sample in the experience replay pool;

[0023] S3.5: Determine whether the round is over, that is, the current node is the end point or the maximum number of actions has been reached. If and ,make , , return to step S3.3; otherwise, go to step S3.6;

[0024] S3.6: Order , judge whether it has reached the update round, if , return to step S3.2; otherwise, go to step S3.7;

[0025] S3.7: Take out all samples in the experience replay pool and update the Actor and Critic neural networks;

[0026] S3.8: Determine whether to end the training. If , return to step S3.2; otherwise, the model training is completed.

[0027] Furthermore, in step S3.3, the action probability distribution is obtained The formula is as follows:

[0028] (2)

[0029] (3)

[0030] Among them, the formula (2) for Next execution The unnormalized probability prediction value of Afterwards, the agent is judged whether it can reach the adjacent node with a movement feasibility of 1. Is it valid? for The unnormalized prediction values ​​of all actions after invalid action masking;

[0031] Furthermore, in step S3.7, the Actor network is updated, specifically to maximize the objective function:

[0032] (4)

[0033] in is a hyperparameter that controls the clipping range. is the probability ratio before and after the strategy update, The function limits the amplitude of the policy update to , avoid major strategic changes. The estimated value of the advantage function at time step t after the advantage function is normalized:

[0034] (5)

[0035] in 、 All in the batch The arithmetic mean and standard deviation of To maintain a small constant for numerical stability, is the estimated value of the advantage function at time step t, calculated by generalized advantage function estimation:

[0036] (6)

[0037] in is the discount factor, , .

[0038] Furthermore, in step S3.7, the critic network is updated, specifically to minimize the function error:

[0039] (7)

[0040] in, Critic before update The predicted value of . BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the process of the photonic on-chip network routing method based on the PPO-GRU algorithm of the present invention;

[0042] Figure 2 A topological diagram of an N×M hexagonal circulable PNoC simulation array provided in an embodiment;

[0043] Figure 3 A schematic diagram of coordinates of a basic optical switch unit in an N×M hexagonal recyclable PNoC provided in an embodiment;

[0044] Figure 4 Schematic diagram of the DRL-based PNoC routing model of the present invention;

[0045] Figure 5 Schematic diagram of the action space of the N×M hexagonal cyclic PNoC routing model provided in the embodiment;

[0046] Figure 6 The figure is a flow chart of the present invention for training a PNoC routing model according to the PPO-GRU algorithm. Specific implementation methods

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0048] This embodiment provides a hexagonal photonic on-chip network routing method based on the PPO-GRU algorithm. Figure 1 , the routing method includes:

[0049] S1: Build an N×M hexagonal recyclable PNoC simulation array, obtain the transmission loss of optical signals under different switching states, and predefine the optical network model based on the topological structure characteristics of the PNoC simulation array;

[0050] In this embodiment, the N×M hexagonal circulable PNoC simulation array is as follows: Figure 2 As shown in Figure 1, the network consists of N rows and M columns of hexagonal grids, each of which contains 6 basic optical switch units and 6 optical waveguide links. To represent the PNoC simulation array, where the node set Represents the set of all basic optical switch units in the network, the edge set Represents the set of all optical waveguide links, loss set Indicates that the optical signal comes from The input is from port p and the output is from port q. The transmission loss set; Each basic optical switch unit has 3 ports, the optical signal can be input into the optical switch from any port and output from the other 2 ports; Each basic optical switch unit is connected to other optical switches through an optical waveguide link. Except for the outer basic optical switch unit connected to two optical switch units, the other optical switch units are connected to three optical switch units; Figure 3 The two-dimensional coordinate system shown is used to represent the position of the optical switch in the network, and the position of the optical switch unit is defined as (x, y).

[0051] S2: Based on the PNoC model in step S1 and the optical signal transmission characteristics, a DRL-based PNoC routing model is built. The routing problem is described as a Markov decision process, and the intelligent agent, state space S, action space A, reward function R, and experience replay pool in reinforcement learning are established.

[0052] In this embodiment, the DRL-based PNoC routing model is as follows: Figure 4 As shown in the figure, the intelligent agent includes an actor and a critic, where the actor is responsible for learning the routing strategy and the critic is responsible for commenting on the quality of the strategy. Both use GRU as the neural network structure, use orthogonal initialization of GRU network parameters, and use tanh activation function.

[0053] The state space consists of three parts: the agent's moving direction, relative coordinates, and movement feasibility; the agent's moving direction like Figure 3 The starting point and the end point are shown as the direction in which the agent moves from the previous node to the current node; the relative coordinates Refers to the axial coordinate difference between the current node and the destination node of the intelligent agent Mobile feasibility Refers to whether it is feasible for the agent to move from the current node to the adjacent node, using 0 / 1 to indicate infeasible / feasible; a one-dimensional tensor is used to encode the state of each time step and define the state space ,like Figure 3 The agent is at the starting state .

[0054] Action Space is the set of all possible transformations of the agent's moving direction. The action space in this embodiment is as follows: Figure 5 As shown, it consists of 12 feasible actions: DR-L, DR-UR, DL-R, DL-UL, L-UR, L-DR, UL-R, UL-DL, UR-L, UR-DR, R-DL, R-UL, coded with 0-11 to define the action space .

[0055] The reward function consists of two parts, one of which is the negative value of the transmission loss that the agent will get after each action, and the other is the arrival reward that the agent will get when it reaches the destination node. When , the reward obtained by the agent is:

[0056] (1)

[0057] in, is a non-zero scaling factor, , or 1, is a positive value associated with the size of the network.

[0058] The experience replay pool stores sample data, and treats the samples generated in each round as a sample block. During each update, all sample blocks are taken out and divided into multiple parts for multiple sampling and training to update network parameters.

[0059] S3: Train the PNoC routing model based on step S2 according to the PPO-GRU algorithm;

[0060] In this embodiment, if Figure 6 The PNoC routing model is trained according to the PPO-GRU algorithm, specifically:

[0061] S3.1: Set the maximum number of iterations , GRU update frequency The maximum number of actions per round , let the current iteration number ;

[0062] S3.2: Randomly initialize the starting point , current node and end point ,in , initialize the environment and get the initial state , initialize the experience replay pool, and make the number of actions of the current round agent ;

[0063] S3.3: Current state In the GRU of the input actor, the unnormalized action probability prediction value output by the GRU is masked for invalid actions to obtain the action probability distribution , Actor selects actions based on probability sampling , environment according to renew And output The rewards , the state at the next moment and whether the destination has been reached ;

[0064] In the embodiment, the action probability distribution is obtained The formula is as follows:

[0065] (2)

[0066] (3)

[0067] Among them, the formula (2) for Next execution The unnormalized probability prediction value of Afterwards, the agent is judged whether it can reach the adjacent node with a movement feasibility of 1. Is it valid? for The unnormalized predicted values ​​of all actions after invalid action masking are used. Using the invalid action masking state can filter out invalid routing operations restricted by the switch state, improve exploration capabilities, and solve the problem that sparse rewards make it difficult for the model to learn the target;

[0068] S3.4: After getting After that, Actor selects the action through action sampling , the agent executes , environment according to Update and output The rewards , the state at the next moment And whether the destination node is reached , here we do not directly Deposit into the experience replay pool, but first use reward scaling to Processing, dynamically calculate the standard deviation of the discount of the total reward in a round , then Divide by the standard deviation to get , then and Critic The value function predicts the value As a sample stored in the experience replay pool, reward scaling can improve the stability of model training;

[0069] S3.5: Determine whether the round is over, that is, the current node is the end point or the maximum number of actions has been reached. If and ,make , , return to step S3.3; otherwise, go to step S3.6;

[0070] S3.6: Order , judge whether it has reached the update round, if , return to step S3.2; otherwise, go to step S3.7;

[0071] S3.7: Take out all samples in the experience replay pool and update the Actor and Critic neural networks;

[0072] In the embodiment, the Actor network is updated to maximize the objective function:

[0073] (4)

[0074] in is a hyperparameter that controls the clipping range. is the probability ratio before and after the strategy update, The function limits the amplitude of the policy update to , avoid major strategic changes. The estimated value of the advantage function at time step t after the advantage function is normalized:

[0075] (5)

[0076] in 、 All in the batch The arithmetic mean and standard deviation of To maintain a small constant for numerical stability, is the estimated value of the advantage function at time step t, calculated by generalized advantage function estimation:

[0077] (6)

[0078] in is the discount factor, , ,Using advantage function normalization can improve the stability of model training and accelerate convergence.

[0079] In the embodiment, the critic network is updated, specifically to minimize the function error:

[0080] (7)

[0081] in, Critic before update The predicted value of .

[0082] After the update is complete, all samples will be deleted from the experience replay pool.

[0083] S3.8: Determine whether to end the training. If , return to step S3.2; otherwise, the model training is completed.

[0084] S4: Using the PNoC routing model trained in step S3, obtain the optical routing path with the minimum loss. In an embodiment, the starting point and end point of the optical signal are input into the routing model, and the model can autonomously determine the optical routing path with the minimum loss from the starting point to the end point and output it.

Claims

1. The purpose of the present invention is to provide a photonic on-chip network routing method based on the PPO-GRU algorithm, characterized in that: include: S1: Build a PNoC simulation array, obtain the transmission loss of optical signals under different switching states, and predefine an optical network model based on the topological structure characteristics of the PNoC simulation array; S2: Based on the PNoC model in step S1 and the optical signal transmission characteristics, a DRL-based PNoC routing model is built. The routing problem is described as a Markov decision process, and the intelligent agent, state space S, action space A, reward function R, and experience replay pool in reinforcement learning are established. S3: Train the PNoC routing model based on step S2 according to the PPO-GRU algorithm; S4: Using the PNoC routing model trained in step S3, obtain the optical routing path with the minimum loss.

2. The photonic on-chip network routing method based on the PPO-GRU algorithm according to claim 1, characterized in that: In step S1, the PNoC simulation array consists of basic optical switch units and optical waveguide links; a simple weighted directed graph is used To represent the PNoC simulation array, where the node set Represents the set of all basic optical switch units in the network, the edge set Represents the set of all optical waveguide links, loss set Indicates that the optical signal comes from The input is from port p and the output is from port q. The transmission loss set is: each basic optical switch unit has m ports, where m ≥ 2. The optical signal can enter the optical switch from any port and exit the optical switch from the remaining m-1 ports. Each basic optical switch unit is connected to other optical switches through an optical waveguide link. The position of the optical switch in the network is represented by a two-dimensional coordinate system, and the position of the optical switch unit is defined as (x, y).

3. The PNoC routing method based on the PPO-GRU algorithm according to claim 1, characterized in that: In step S2, the state space consists of three parts: the agent's movement direction, relative coordinates, and movement feasibility; The direction of movement of the intelligent agent The direction in which the agent moves from the previous node to the current node; relative coordinates Refers to the axial coordinate difference between the current node and the destination node of the intelligent agent Mobile feasibility Refers to whether it is feasible for the agent to move from the current node to the adjacent node, using 0 / 1 to indicate infeasible / feasible; a one-dimensional tensor is used to encode the state of each time step and define the state space ; Action Space is the set of all possible changes in the agent's moving direction; the reward function consists of two parts, one of which is the negative value of the transmission loss obtained by the agent after each action, and the other is the arrival reward that the agent will obtain when it reaches the destination node.

4. The PNoC routing method based on the PPO-GRU algorithm according to claim 1, characterized in that: In step S3, the PNoC routing model is trained according to the PPO-GRU algorithm, including: S3.1: Set the maximum number of iterations , GRU update frequency The maximum number of actions per round , let the current iteration number ; S3.2: Randomly initialize the starting point , current node and end point ,in , initialize the environment and get the initial state , initialize the experience replay pool, and make the number of actions of the current round agent ; S3.3: Current state In the GRU of the input actor, the unnormalized action probability prediction value output by the GRU is masked for invalid actions to obtain the action probability distribution , Actor selects actions based on probability sampling , environment according to renew And output The rewards , the state at the next moment and whether the destination has been reached ; S3.4: Use reward scaling Processing, dynamically calculate the standard deviation of the discount of the total reward in a round , then Divide by the standard deviation to get ;Will and Critic The value function predicts the value Stored as a sample in the experience replay pool; S3.5: Determine whether the round is over, that is, the current node is the end point or the maximum number of actions has been reached. If and ,make , , return to step S3.3; otherwise, go to step S3.6; S3.6: Order , judge whether it has reached the update round, if , return to step S3.2; otherwise, go to step S3.7; S3.7: Take out all samples in the experience replay pool and update the Actor and Critic neural networks; S3.8: Determine whether to end the training. If , return to step S3.2; otherwise, the model training is completed.

5. The method of training a PNoC routing model according to the PPO-GRU algorithm according to claim 4, wherein: In step S3.3, the unnormalized action probability prediction value output by GRU is masked for invalid actions to obtain the action probability distribution The calculation formula is as follows: (1) (2) Among them, the formula (1) for Next execution The unnormalized probability prediction value of Afterwards, the agent is judged whether it can reach the adjacent node with a movement feasibility of 1. Is it effective? for The unnormalized prediction values ​​of all actions after invalid action masking.

6. The method of training a PNoC routing model according to the PPO-GRU algorithm according to claim 4, wherein: In step S3.7, the formula for updating the Actor network is as follows: (3) in is a hyperparameter that controls the clipping range. is the probability ratio before and after the strategy update, The function limits the amplitude of the policy update to , avoid major strategic changes. The estimated value of the advantage function at time step t after the advantage function is normalized: (4) in 、 All in the batch The arithmetic mean and standard deviation of To maintain a small constant for numerical stability, is the estimated value of the advantage function at time step t, calculated by generalized advantage function estimation: (5) in is the discount factor, , .

7. The method of training a PNoC routing model according to the PPO-GRU algorithm according to claim 4, wherein: In step S3.7, the formula for updating the Critic network is as follows: (6) in, Critic before update The predicted value of .

Images