Adaptive routing method and system based on Motif statistics and reinforcement learning

Abstract

The invention provides a self-adaptive routing method and system based on Motif statistics and reinforcement learning, an on-chip routing network (Network-on-Chip, NoC) provides a communication basis for a decentralized many-core chip architecture, and the performance of the NoC is closely related to the routing method; according to the adaptive routing method and system, the data communication path is dynamically adjusted according to the actual operation scene of the chip, and it can be guaranteed that an on-chip routing network can express efficient communication performance in different complex communication scenes. Real-time global communication information is processed based on a complex network analysis method Motif, a Motif analysis result and other routing related information are combined, and a path selection strategy is trained through a deep reinforcement learning algorithm. The adaptive routing algorithm based on global communication analysis has stronger adaptive ability, and can provide better data communication time delay and higher data throughput in a complex and dynamically changing scene.

Description

Technical Field

[0001] This invention relates to the technical field of distributed communication systems, and more specifically, to an adaptive routing method and system based on Motif statistics and reinforcement learning. Background Technology

[0002] With the advancement of integrated circuit technology and the demands of productivity development, traditional single-core chips can no longer meet application requirements. Many-core chips integrate an increasing number of processor cores, sometimes reaching hundreds or even thousands. Data communication has become a key bottleneck for chip performance. Traditional bus-based communication architectures suffer from bottlenecks in scalability, bandwidth, and latency. Networked on-chip routing networks can effectively solve these problems. As the core module determining the data transmission path, the routing algorithm plays a crucial role in the performance of on-chip routing networks. Traditional fixed routing algorithms struggle to effectively handle dynamically changing communication loads in the network, easily leading to congestion and degraded system performance. Therefore, designing efficient adaptive routing algorithms to dynamically adjust routing strategies, avoid congestion, and improve communication efficiency has become an important research topic in on-chip network design.

[0003] Adaptive routing technology allows routers to flexibly select paths based on current network conditions (such as traffic, congestion, and latency). Compared to static routing, adaptive routing can dynamically optimize communication paths, reduce congestion, and improve throughput. However, existing adaptive routing algorithms are mostly based on local information and struggle to comprehensively analyze global communication patterns within the network, thus potentially failing to achieve optimal performance in complex network environments.

[0004] A patent document with publication number CN116614437A discloses an adaptive QoS intelligent routing method based on deep reinforcement learning, including the following steps: establishing a neural network; collecting routing samples; and training the neural network. This adaptive routing algorithm relies heavily on local information and struggles to comprehensively analyze global communication patterns within the network, thus potentially failing to achieve optimal performance in complex network environments.

[0005] Therefore, a new technical solution is needed to improve the above-mentioned technical problems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide an adaptive routing method and system based on Motif statistics and reinforcement learning.

[0007] An adaptive routing method based on Motif statistics and reinforcement learning, provided by the present invention, includes the following steps:

[0008] Step S1: Design a routing calculation strategy to output an optional output port when a data packet arrives at a new routing node;

[0009] Step S2: Run the on-chip routing network, obtain the optional output ports by the routing calculation strategy, and count the downstream routing credit information of the optional ports. Randomly select a port for data output.

[0010] Step S3: Statistically analyze global communication status information at specified intervals;

[0011] Step S4: Analyze the global communication status information using Motif and output the analysis results;

[0012] Step S5: Based on the Motif analysis results and downstream Credit information as input, train a deep reinforcement learning network offline as a routing strategy;

[0013] Step S6: Fit the deep reinforcement learning network based on the tree model and simplify it into combinational circuit logic;

[0014] Step S7: Deploy the combinational circuit logic obtained in step S6 into the chip's on-chip router to form an on-chip routing network. The routing calculation strategy outputs selectable output ports, and the combinational logic circuit dynamically adjusts the output ports according to real-time communication status information.

[0015] Preferably, the routing calculation strategy in step S1 is a fully adaptive routing calculation strategy based on Duato's theorem and virtual channel routing architecture; the fully adaptive routing calculation strategy provides at most two optional output ports each time, and then combines the routing selection strategy to select one of the optional output ports as the final output.

[0016] Preferably, the downstream routing credit information in step S2 is the number of idle virtual channels downstream.

[0017] Preferably, the global communication status information in step S3 is a data communication matrix. Whenever there is data communication between two routing nodes, the corresponding matrix position is recorded as 1, and the communication matrix is ​​output and reset every specified period.

[0018] Preferably, the analysis result in step S4 is the number of each of the six three-node Motifs existing in the communication matrix; the three-node Motif is a special case of the Motif analysis method for directed graph networks containing three nodes, and there are a total of 13 Motif paradigms.

[0019] Preferably, the deep reinforcement learning network in step S5 is DQN (Deep Q-network).

[0020] Preferably, the tree model in step S6 is a decision tree, and its execution process corresponds to the combinational logic on the hardware circuit, which can be completed in a single cycle.

[0021] Preferably, the on-chip router in step S7 is a single node of the on-chip routing network, and all nodes together constitute the entire communication network.

[0022] This invention also provides an adaptive routing system based on Motif statistics and reinforcement learning, the system comprising the following modules:

[0023] Module M1: Designs routing calculation strategies and outputs an optional output port when a data packet arrives at a new routing node;

[0024] Module M2: Runs the on-chip routing network, obtains the selectable output ports by the routing calculation strategy, counts the downstream routing credit information of the selectable ports, and randomly selects a port for data output;

[0025] Module M3: Collects global communication status information at specified intervals;

[0026] Module M4: Analyzes global communication status information using Motif and outputs the analysis results;

[0027] Module M5: Based on the Motif analysis results and downstream Credit information as input, a deep reinforcement learning network is trained offline as a routing strategy;

[0028] Module M6: Fits deep reinforcement learning networks based on tree models, simplifying them into combinational circuit logic;

[0029] Module M7: Deploys the combinational circuit logic obtained from Module M6 into the chip's on-chip router to form an on-chip routing network. The routing calculation strategy outputs selectable output ports, and the combinational logic circuit dynamically adjusts the output ports according to real-time communication status information.

[0030] Preferably, the routing calculation strategy in module M1 is a fully adaptive routing calculation strategy based on Duato's theorem and virtual channel routing architecture; the fully adaptive routing calculation strategy provides at most two optional output ports each time, and subsequently combines with the routing selection strategy to select one of the optional output ports as the final output;

[0031] The downstream routing credit information in module M2 is the number of idle virtual channels downstream;

[0032] The global communication status information in module M3 is a data communication matrix. Whenever there is data communication between two routing nodes, the corresponding matrix position is recorded as 1. The communication matrix is ​​output and reset every specified period.

[0033] The analysis results in module M4 are the number of each of the six three-node Motifs that exist in the communication matrix; the three-node Motif is a special case of directed graph networks containing three nodes in the Motif analysis system, and there are a total of 13 Motif paradigms.

[0034] The deep reinforcement learning network in module M5 is DQN (Deep Q-network);

[0035] The tree model in module M6 is a decision tree, and its execution process corresponds to the combinational logic on the hardware circuit, which can be completed in a single cycle.

[0036] The on-chip router in module M7 is a single node in the on-chip routing network, and all nodes together constitute the entire communication network.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. This invention combines Motif statistics and reinforcement learning techniques to dynamically identify high-frequency communication patterns in a network, adjust routing paths in real time, reduce congestion, and improve data transmission efficiency.

[0039] 2. The system of the present invention utilizes reinforcement learning to continuously optimize the routing strategy, enabling it to maintain efficient routing selection under different network loads and communication requirements, thereby improving the system's throughput and latency performance;

[0040] 3. This invention can be effectively expanded in multi-core systems, adapt to the complex communication needs of large-scale on-chip networks, and maintain a good balance between power consumption and performance;

[0041] 4. Through this invention, on-chip routing networks can achieve more efficient communication and adapt to the future trend of large-scale chip integration, providing reliable many-core communication support for high-performance computing, artificial intelligence and other fields. Attached Figure Description

[0042] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0043] Figure 1 This is a schematic diagram of the overall structure of the adaptive routing method and system of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating the connection of 13 possible three-node Motifs according to the present invention;

[0045] Figure 3 This is a schematic diagram illustrating the calculation formulas for the six types of three-node Motifs involved in this invention;

[0046] Figure 4 This is a schematic diagram of the training framework of the deep reinforcement learning network of the present invention;

[0047] Figure 5 This is a schematic diagram of the hardware circuit for Motif calculation in this invention. Detailed Implementation

[0048] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0049] Example 1:

[0050] Reference Figure 1 and Figure 2 According to the present invention, an adaptive routing method based on Motif statistics and reinforcement learning is provided, the method comprising the following steps:

[0051] Step S1: Design a routing calculation strategy to output an optional output port when a data packet arrives at a new routing node; the routing calculation strategy is a fully adaptive routing calculation strategy based on Duato's theorem and the virtual channel routing architecture; the fully adaptive routing calculation strategy provides at most two optional output ports each time, and subsequently combines with the routing selection strategy to select one of the optional output ports as the final output.

[0052] Step S2: Run the on-chip routing network, obtain the selectable output ports by the routing calculation strategy, and count the downstream routing credit information of the selectable ports. Randomly select a port for data output; the downstream routing credit information is the number of idle virtual channels downstream.

[0053] Step S3: Statistically analyze global communication status information at specified intervals; global communication status information is a data communication matrix. Whenever there is data communication between two routing nodes, the corresponding matrix position is recorded as 1. Output and reset the communication matrix at specified intervals.

[0054] Step S4: Analyze the global communication state information using Motif and output the analysis results; the analysis results are the number of each of the six three-node Motifs existing in the communication matrix; the three-node Motif is a special case of directed graph networks containing three nodes in the Motif analysis method, and there are a total of 13 Motif paradigms.

[0055] Step S5: Based on the Motif analysis results and downstream Credit information as input, train a deep reinforcement learning network offline as a routing strategy; the deep reinforcement learning network is DQN (Deep Q-network).

[0056] Step S6: Fit the deep reinforcement learning network based on the tree model and simplify it into combinational circuit logic; the tree model is a decision tree, and its execution process corresponds to the combinational logic on the hardware circuit, which can be completed in a single cycle.

[0057] Step S7: Deploy the combinational logic obtained in Step S6 into the chip's on-chip router to form an on-chip routing network. Optional output ports are output based on the routing calculation strategy, and the combinational logic dynamically adjusts the output ports according to real-time communication status information. The on-chip router is a single node in the on-chip routing network; all nodes together constitute the entire communication network.

[0058] The present invention also provides an adaptive routing system based on Motif statistics and reinforcement learning. The adaptive routing system based on Motif statistics and reinforcement learning can be implemented by executing the process steps of the adaptive routing method based on Motif statistics and reinforcement learning. That is, those skilled in the art can understand the adaptive routing method based on Motif statistics and reinforcement learning as a preferred embodiment of the adaptive routing system based on Motif statistics and reinforcement learning.

[0059] Example 2:

[0060] This invention also provides an adaptive routing system based on Motif statistics and reinforcement learning, the system comprising the following modules:

[0061] Module M1: Designs a routing calculation strategy that outputs an optional output port when a data packet arrives at a new routing node; the routing calculation strategy is a fully adaptive routing calculation strategy based on Duato's theorem and the virtual channel routing architecture; the fully adaptive routing calculation strategy provides at most two optional output ports each time, and subsequently combines with a routing selection strategy to select one of the optional output ports as the final output;

[0062] Module M2: Runs the on-chip routing network, obtains the selectable output ports based on the routing calculation strategy, and counts the downstream routing credit information of the selectable ports. It then randomly selects a port for data output. The downstream routing credit information represents the number of idle virtual channels downstream.

[0063] Module M3: Calculates global communication status information at specified intervals; the global communication status information is a data communication matrix. Whenever there is data communication between two routing nodes, the corresponding matrix position is recorded as 1. The communication matrix is ​​output and reset at specified intervals.

[0064] Module M4: Analyzes global communication state information using Motif and outputs the analysis results; the analysis results are the number of each of the six three-node Motifs existing in the communication matrix; the three-node Motif is a special case of directed graph networks containing three nodes in the Motif analysis system, and there are a total of 13 Motif paradigms.

[0065] Module M5: Based on the Motif analysis results and downstream Credit information as input, an offline deep reinforcement learning network is trained as a routing strategy; the deep reinforcement learning network is DQN (Deep Q-network);

[0066] Module M6: Fits deep reinforcement learning networks based on tree models, simplifying them into combinational circuit logic; the tree model is a decision tree, and its execution process corresponds to combinational logic on hardware circuits, which can be completed in a single cycle;

[0067] Module M7: Deploys the combinational circuit logic obtained from Module M6 into the chip's on-chip router to form an on-chip routing network. The routing calculation strategy outputs selectable output ports, and the combinational logic circuit dynamically adjusts the output ports according to real-time communication status information. The on-chip router is a single node in the on-chip routing network, and all nodes together constitute the entire communication network.

[0068] Example 3:

[0069] This invention designs an adaptive routing method and system based on Motif statistical methods and reinforcement learning. Network on Chip (NoC) provides the communication foundation for decentralized many-core chip architectures, and its performance is closely related to the routing method. The adaptive routing method and system dynamically adjust the data communication path according to the actual operating scenario of the chip, ensuring that the NoC can exhibit high-efficiency communication performance under various complex communication scenarios. This invention uses the complex network analysis method Motif to process real-time global communication information, and combines the Motif analysis results with other routing-related information to train a path selection strategy using a deep reinforcement learning algorithm. The adaptive routing algorithm based on global communication analysis has stronger adaptability, providing better data communication latency and higher data throughput in complex and dynamically changing scenarios. Furthermore, this invention also uses a tree model to fit a deep reinforcement learning neural network, converting it into simplified circuit logic to ensure the smooth deployment and operation of the designed routing method and system on mobile chips. The adaptive routing method and system of this invention are applicable to mobile AI processors based on many-core architectures, with application scenarios including autonomous driving, autonomous robots, and other artificial intelligence devices. This invention implements a high-performance global adaptive routing method through the complex directed network analysis method Motif and deep reinforcement learning algorithms. Furthermore, it reduces the execution overhead of the global adaptive routing method by using tree model fitting, thus facilitating the practical on-chip deployment of the designed routing method and system. This invention can significantly improve the overall system's communication efficiency, reduce communication latency, and enhance the system's adaptability, enabling it to be applied to various complex scenarios.

[0070] An adaptive routing method and system based on Motif statistics and reinforcement learning includes the following steps:

[0071] Step 1: Design a routing calculation strategy that outputs an optional output port when a data packet arrives at a new routing node; the routing calculation strategy is a fully adaptive routing calculation strategy based on Duato's theorem and the virtual channel routing architecture.

[0072] Step 2: Run the on-chip routing network, obtain the selectable output ports by the routing calculation strategy, and count the downstream routing credit information of the selectable ports. Randomly select a port for data output; the downstream routing credit information is the number of idle virtual channels downstream.

[0073] Step 3: Statistically analyze global communication status information at specified intervals; global communication status information is a data communication matrix. Whenever there is data communication between two routing nodes, the corresponding matrix position is recorded as 1. Output and reset the communication matrix at specified intervals.

[0074] Step 4: Analyze the global communication status information using Motifs and output the analysis results; the analysis results are the number of each of the six possible three-node Motifs that exist in the communication matrix.

[0075] Step 5: Using the Motif analysis results and downstream Credit information as input, train a deep reinforcement learning network offline as a routing strategy; the deep reinforcement learning network is DQN (Deep Q-network).

[0076] Step 6: Fit the deep reinforcement learning network based on the tree model and simplify it into combinational circuit logic; the tree model is a decision tree, and its execution process corresponds to simple combinational logic on the hardware circuit, which can be completed in a single cycle.

[0077] Step 7: Deploy the combinational logic obtained in Step 6 into the chip's on-chip router to form an on-chip routing network. The routing calculation strategy outputs selectable output ports, and the combinational logic dynamically adjusts the output ports based on real-time communication status information. The on-chip router is a single node in the on-chip routing network; numerous nodes together constitute the entire communication network. This invention provides an adaptive routing method for on-chip routers, including an adaptive routing calculation strategy based on existing achievements and an adaptive routing selection strategy based on Motif statistics and reinforcement learning proposed in this invention. Both work together to ensure correct and efficient data transmission.

[0078] Adaptive routing is manifested in two aspects: adaptive route calculation strategy and adaptive route selection strategy. This invention mainly provides the adaptive route selection strategy. The adaptive route calculation strategy provides at most two optional output ports at a time. Subsequently, it needs to be combined with the route selection strategy to select one of the optional output ports as the final output.

[0079] The three-node Motif is a special case of directed graph networks containing three nodes in the Motif analysis method. There are 13 possible Motif paradigms. However, the connections of on-chip routing networks are relatively regular, so this invention only focuses on the six three-node Motif paradigms that may exist in the statistical communication matrix.

[0080] This invention achieves efficient data communication in complex scenarios by dynamically adjusting data transmission paths. It aims to improve the communication efficiency, resource utilization, and robustness of distributed systems, and is applicable to applications such as multi-core processors, heterogeneous computing, embedded systems, and IoT devices. This invention introduces Motif analysis and reinforcement learning techniques. Motif extracts frequently occurring communication patterns in the network by statistically analyzing global communication state information, revealing the structure and communication patterns of on-chip networks, thus providing important basis for routing decisions. Reinforcement learning, through continuous trial and adjustment, learns the optimal routing strategy, effectively responding to dynamic changes in network state and further enhancing the router's decision-making capabilities. The adaptive routing method based on Motif statistics and reinforcement learning is expected to significantly improve the performance of on-chip networks, providing a more intelligent and efficient communication solution for on-chip multi-core systems.

[0081] This invention provides an adaptive routing method and system based on Motif statistics and reinforcement learning, aiming to improve the communication efficiency and performance stability of on-chip routing networks. This method combines Motif statistical analysis of high-frequency communication patterns in the network with the self-learning capability of reinforcement learning algorithms to dynamically adjust routing strategies and optimize data transmission paths, thereby effectively addressing the limitations of traditional routing algorithms in complex network conditions. First, this invention uses Motif statistical technology to analyze the traffic distribution and communication characteristics in the on-chip network in real time, extracting common communication patterns. Based on these patterns, the system uses reinforcement learning technology to train and construct a highly adaptive routing strategy system. This enables the router to continuously adjust its routing strategy according to dynamic changes in network conditions during operation, adapting to varying communication needs.

[0082] like Figures 1-5 As shown, this invention provides an adaptive routing method and system based on Motif statistics and reinforcement learning. Specifically, this invention uses the complex network analysis method Motif to process real-time global communication information, and combines the Motif analysis results with other routing-related information to train a path selection strategy using a deep reinforcement learning algorithm. The adaptive routing algorithm based on global communication analysis has stronger adaptability and can provide better data communication latency and higher data throughput in complex and dynamically changing scenarios. In addition, this invention also fits the deep reinforcement learning neural network with a tree model, converting it into simplified circuit logic to ensure the smooth deployment and operation of the designed routing method and system on mobile terminal chips.

[0083] Figure 1The overall architecture of the adaptive routing method and system of this invention is demonstrated. Taking a 3x3 scale as an example, the left side shows an on-chip routing network based on a mesh topology. During operation, the Global State Memory (GSM) stores global communication state information; specifically, the GSM stores the global communication matrix W. When R... i To R j When data transmission occurs, W will be... i,j Set to 1, global state information every T m The Motif Calculation (MC) circuit updates once per operating cycle. Based on the global communication information matrix stored in GSM, the MC calculates the quantity of each of the six three-node Motifs and stores the results in the Motif Weight Memory.

[0084] There are 13 possible structures for a three-node Motif, such as Figure 2 As shown, however, in a mesh topology, there are no pairs of interconnected nodes between the three nodes. Therefore, this invention only focuses on the M nodes that may exist in the communication matrix. 1-6 Motifs of type m 1-6 The result can be obtained from... Figure 3 The motif calculation formula shown is obtained, where U=W⊙W T Let U be a matrix consisting of unidirectional communication edges. i,j A value of 1 indicates that in T m Inner R i To R j There is data transmission, and R j To R i No data transmission; similarly, B = WU is a matrix formed by bidirectional communication edges, if B i,j A value of 1 indicates that in T m Inner R i To R j and R j To R i All have data transmission; N corresponds to a matrix consisting of nodes without communication connections, if N i,j A value of 1 indicates that in T m Inner R i and R j There was no data transmission between them. For further explanation... Figure 3 The formula shown is now defined. R represents i To R j For one-way communication, R represents i and R j It is a two-way communication. R represents i and R j There is no data communication between them. Taking the calculation of m1 as an example, such as... Figure 2 As shown, counting the number of M1 nodes in a directed graph is equivalent to counting the number of one-way edges from node A to node B. The number of nodes, and the third node C must satisfy: and That is U A,C and N C,B All are 1. Since all the above matrices are binary, matrix multiplication can identify and count the corresponding connection combinations. Specifically, U×N identifies combinations that satisfy... and The node connection combination, in order to ensure The calculation result needs to be multiplied by U to obtain matrix R, where the non-zero elements correspond to nodes A and B that satisfy the connectivity relationships in M1. In the above calculation, since M1... and They are equivalent, therefore there is double counting, where m1 is the number of all non-zero elements in R divided by 2.

[0085] Leveraging the connectivity characteristics of mesh topology, the above calculation process can be further simplified. Assuming matrix X is an identity matrix with diagonal elements of 0 (nodes cannot communicate with themselves, hence diagonal elements are 0), then N can be represented as XW. Therefore, R = (U × (XW))⊙U = (U × X)⊙U - (U × W)⊙U. Considering (U × W)⊙U, it needs to satisfy... and (or ),as well as However, this connection scenario is impossible because there is no pairwise data communication between the three nodes in a mesh topology. Therefore, the calculation of m1 can be simplified to SUM((U×X)⊙U) / 2. That is, it needs to satisfy... and And since B and C are not the same node (corresponding to the diagonal elements in matrix X not being 1), for ease of subsequent judgment, GSM stores communication information in matrix format. Considering that a router in a mesh topology can communicate with a maximum of four other routers (northwest, southeast), each row of GSM data corresponds to one router, and each row contains four 2-bit data entries, corresponding to the connection status of that router with its neighboring routers. 1,0 A value of 0 indicates that R1 has no communication connection with its western neighbor router (i.e., R0), a value of 1 indicates that R1 has a one-way communication connection to R0, and a value of 2 indicates that the two have a two-way communication connection. With the help of this storage format of GSM, the connection conditions can be easily determined and the value of m1 can be calculated.

[0086] m 1-6 The corresponding computational logic in the hardware circuit is as follows: Figure 5 As shown, 'a' represents the position ID of node A in the NoC topology, and 'index0' is the position of node B relative to node A (northwest, southeast). It stores the communication information between node A and node B. Similarly, index1 is the position of node C relative to node A. The system stores the communication information between nodes A and C. During computation, it needs to iterate through all possible values ​​of 'a', 'index0', and 'index1'. In a 3x3 mesh topology, the possible values ​​of 'a' are 0-8, and the possible values ​​of 'index0' and 'index1' are 0-3. The system uses the communication information stored in the GSM to determine if a specified operator, such as U×X (UX), is satisfied. If satisfied, the GSM outputs an enable signal, incrementing the corresponding operator value (i.e., incrementing the counter). After looping through all possible input values, the value of 'm1' is output and stored in the MVM. Figure 1 The core is a virtual tunnel router architecture based on a wormhole routing switching strategy, with multiple routers interconnected to form an overall on-chip routing network. Each router contains a write signal to GSM and a read signal from MWM. 1-6 The read signal; Figure 1 The right side shows the execution flow of the Motif routing module, which receives m 1-6 The tree model selects the output port by taking downstream Credit information as input.

[0087] The tree model is a combinational logic fitted by a deep reinforcement learning network trained offline. Reinforcement learning (RL) aims to maximize long-term rewards (typically average packet latency in NoC), accumulating experience and learning feedback through long-term training, demonstrating its advantage in enhancing decision-making capabilities even in unfamiliar environments. In this invention, each router acts as an agent, and RL optimizes routing strategies through offline training. During runtime, RL selects the optimal action (i.e., output port) based on a defined state vector, maximizing the reward based on a reward function. To ensure the feasibility of the hardware circuitry, this invention employs a deep Q-network (DQN), the overall framework of which is as follows: Figure 4 As shown. The goal of RL training is to learn the optimal action selection policy π, denoted as Q(s,a;w) in DQN, where w represents the neural network weights. DQN uses a quadruple (s...a...w ...w...w...w...w...w...w...w...w...w...w...w...w...w...w...w... t ,a t ,r t ,s t+1 Training is performed using a transition. In each step, given the current state s... tBased on a greedy strategy, select an action 'a' from the action space. t Then, perform the action within the environment (i.e., NoC) to obtain an immediate reward r. t And transition to the new state s t+1 ① Traditional DQN typically discards transitions after gradient updates, leading to a waste of experience. Furthermore, transitions corresponding to continuous state changes exhibit strong correlations, which hinders network convergence. To accelerate DQN convergence and improve performance, this invention employs an experience replay technique, storing transitions in a replay buffer. ② During each iteration, a batch of transitions is randomly sampled from the experience replay buffer. ③ Then, mini-batch gradient descent is used to update the network weights. Additionally, we use the target network Q(s,a;w)... - To ensure stability during training, the original network is called the evaluation network Q(s,a;w). Both networks have the same structure but different weight parameters. The loss function L is calculated based on the transition and the following formula ④.

[0088]

[0089] Where y t This represents the target value. Gradient descent is used to update the weights w⑤ during training, where N is the mini-batch size. Only the weights of the evaluation network are adjusted throughout the training process, while the target network is updated according to the specified update frequency T. u The weights of the evaluation network are periodically replicated. This method ensures the relative stability of the target value within each update cycle, thus enhancing the stability of the learning process.

[0090] The training process described above includes the following key variables: 1. Input state variable s t As mentioned above, this invention chooses to use Motif to calculate the result m. 1-6 And the Credit information of two optional ports as state variables; 2. Action variable a t In adaptive routing strategies, there are two possible actions: selecting ports in the X or Y direction; 3. Reward r t The reward setting is crucial for the performance of reinforcement learning. To improve global communication, this invention periodically calculates the global communication latency and uses it as r. t After long-term training, the DQN network will choose a method that maximizes r. t The selection strategy involves randomly providing inputs to the trained network and outputting actions from the DQN. This output serves as the training dataset for the decision tree model, leading to an adaptive routing strategy based on combinational logic.

[0091] Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0092] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0093] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. An adaptive routing method based on Motif statistics and reinforcement learning, characterized in that, The method includes the following steps: Step S1: Design a routing calculation strategy to output an optional output port when a data packet arrives at a new routing node; Step S2: Run the on-chip routing network, obtain the optional output ports by the routing calculation strategy, and count the downstream routing credit information of the optional ports. Randomly select a port for data output. Step S3: Statistically analyze global communication status information at specified intervals; Step S4: Analyze the global communication status information using Motif and output the analysis results; Step S5: Based on the Motif analysis results and downstream Credit information as input, train a deep reinforcement learning network offline as a routing strategy; Step S6: Fit the deep reinforcement learning network based on the tree model and simplify it into combinational circuit logic; Step S7: Deploy the combinational circuit logic obtained in step S6 into the chip's on-chip router to form an on-chip routing network. The routing calculation strategy outputs selectable output ports, and the combinational logic circuit dynamically adjusts the output ports according to real-time communication status information.

2. The adaptive routing method based on Motif statistics and reinforcement learning according to claim 1, characterized in that, The routing calculation strategy in step S1 is a fully adaptive routing calculation strategy based on Duato's theorem and virtual channel routing architecture. The fully adaptive routing calculation strategy provides at most two optional output ports each time, and then combines them with a routing selection strategy to select one of the optional output ports as the final output.

3. The adaptive routing method based on Motif statistics and reinforcement learning according to claim 1, characterized in that, The downstream routing credit information in step S2 is the number of idle virtual channels downstream.

4. The adaptive routing method based on Motif statistics and reinforcement learning according to claim 1, characterized in that, The global communication status information in step S3 is a data communication matrix. Whenever there is data communication between two routing nodes, the corresponding matrix position is recorded as 1. The communication matrix is ​​output and reset every specified period.

5. The adaptive routing method based on Motif statistics and reinforcement learning according to claim 1, characterized in that, The analysis results in step S4 are the number of each of the six three-node Motifs existing in the communication matrix; the three-node Motif is a special case of directed graph networks containing three nodes in the Motif analysis method, and there are a total of 13 Motif paradigms.

6. The adaptive routing method based on Motif statistics and reinforcement learning according to claim 1, characterized in that, The deep reinforcement learning network in step S5 is DQN (Deep Q-network).

7. The adaptive routing method based on Motif statistics and reinforcement learning according to claim 1, characterized in that, The tree model in step S6 is a decision tree, and its execution process corresponds to the combinational logic on the hardware circuit, which can be completed in a single cycle.

8. The adaptive routing method based on Motif statistics and reinforcement learning according to claim 1, characterized in that, The on-chip router in step S7 is a single node in the on-chip routing network, and all nodes together constitute the entire communication network.

9. An adaptive routing system based on Motif statistics and reinforcement learning, characterized in that, The system includes the following modules: Module M1: Designs routing calculation strategies and outputs optional output ports when a data packet arrives at a new routing node; Module M2: Runs the on-chip routing network, obtains the selectable output ports by the routing calculation strategy, counts the downstream routing credit information of the selectable ports, and randomly selects a port for data output; Module M3: Collects global communication status information at specified intervals; Module M4: Analyzes global communication status information using Motif and outputs the analysis results; Module M5: Based on the Motif analysis results and downstream Credit information as input, a deep reinforcement learning network is trained offline as a routing strategy; Module M6: Fits deep reinforcement learning networks based on tree models, simplifying them into combinational circuit logic; Module M7: Deploys the combinational circuit logic obtained from Module M6 into the chip's on-chip router to form an on-chip routing network. The routing calculation strategy outputs selectable output ports, and the combinational logic circuit dynamically adjusts the output ports according to real-time communication status information.

10. The adaptive routing system based on Motif statistics and reinforcement learning according to claim 9, characterized in that, The routing calculation strategy in module M1 is a fully adaptive routing calculation strategy based on Duato's theorem and virtual channel routing architecture. The fully adaptive routing calculation strategy provides at most two optional output ports each time, and then combines the routing selection strategy to select one of the optional output ports as the final output. The downstream routing credit information in module M2 is the number of idle virtual channels downstream; The global communication status information in module M3 is a data communication matrix. Whenever there is data communication between two routing nodes, the corresponding matrix position is recorded as 1. The communication matrix is ​​output and reset every specified period. The analysis results in module M4 are the number of each of the six three-node Motifs that exist in the communication matrix; the three-node Motif is a special case of directed graph networks containing three nodes in the Motif analysis system, and there are a total of 13 Motif paradigms. The deep reinforcement learning network in module M5 is DQN (Deep Q-network); The tree model in module M6 is a decision tree, and its execution process corresponds to the combinational logic on the hardware circuit, which can be completed in a single cycle. The on-chip router in module M7 is a single node in the on-chip routing network, and all nodes together constitute the entire communication network.

Images