Q-learning-based energy-efficient task-driven dynamic on-chip interconnect communication method

Abstract

A Q-learning-based energy-efficient task-driven dynamic on-chip interconnect communication method, relating to the field of on-chip interconnect of computer systems. In the present invention, an adaptive topology is used to map tasks, and on the basis of different features of the tasks, different regions are allocated to perform task processing. In the method, a specific on-chip resource and topology type can be allocated for each task, thereby reducing the consumption of on-chip resources. By means of the adaptive topology, a system can flexibly adjust connection relationships and routing modes in a network-on-chip according to the requirements and characteristics of the tasks. In addition, by allocating different regions, the processing procedures of different tasks can be isolated, thereby reducing the occurrence of interference and conflicts to the greatest extent. A Q-learning reinforcement learning algorithm is used to dynamically change the transmission direction of a routing algorithm and the number of virtual channels during task running using low latency and low power consumption as objectives, so that the balance between performance and power consumption in a network-on-chip can be effectively achieved.

Description

A high-energy-efficiency task-driven dynamic on-chip interconnect communication method based on Q-learning Technical Field

[0001] This invention belongs to the field of on-chip interconnection in computer architecture, and specifically relates to an adaptive on-chip network scheme based on Q-learning reinforcement learning. Background Technology

[0002] Modern processors typically have multiple cores, and on-chip networks (Networks-on-Chip) play a crucial role in multi-processor systems and multi-processor on-chip systems. They solve the problem of communication between cores in multi-core processors, providing low-latency, high-bandwidth, and scalable communication channels. However, current research often shows that on-chip network resources are underutilized, resulting in low resource utilization. During task mapping, load balancing among different core resources needs to be considered to avoid performance loss due to idle core resources. Furthermore, on-chip networks lack targeted topology selection schemes for different tasks. The on-chip network topology determines the physical layout and connections between router nodes and transmission channels, directly affecting message routing and thus network latency. The number of virtual channels in a router also significantly impacts latency and power consumption.

[0003] Network-on-a-chip (NAT) consists of router nodes and transmission channels. The choice of topology has a significant impact on the transmission performance and power consumption of NAT. Topology determines the number of hops or routers a message passes through, thus affecting network latency. Since message transmission through routers and links consumes energy, the impact of topology on hop count directly affects system power consumption. Topology is a crucial part of NAT design, significantly influencing the overall performance and overhead of the NAT. Topology determines the physical distribution of NAT nodes and links and their connection relationships, as well as the number of hops a message can transmit. Therefore, topology design has a substantial impact on transmission latency and power consumption. Currently, directly connected topologies in NAT include ring networks, two-dimensional mesh networks, and two-dimensional Torus networks. Non-directly connected topologies include butterfly and tree structures. Different topologies are suitable for different types of tasks to transmit data. In previous studies on on-chip network interconnection structures, all tasks often used a single topology connection method. This resulted in additional overhead and wasted power consumption for some tasks because they did not need such complex wiring, while for other tasks, the excessive number of message transmission hops led to increased latency and reduced performance.

[0004] The topology of a network-on-a-chip (NIC) directly influences the choice of routing algorithms. NIC routing algorithms are generally classified into three categories: deterministic routing, oblivious routing, and adaptive routing. Deterministic routing is a basic routing method where the route path is fixed for a given source and destination, such as XY routing. Oblivious routing determines the path based on the source and destination nodes, but does not consider real-time or historical network congestion information. Therefore, even if some paths become congested, the route selection will not change. Adaptive routing is a more complex routing strategy that can dynamically change the route path based on network conditions at runtime. In weighted routing, each path or link has a weight, and route selection depends on these weights. Weights can be set based on various factors, such as path length, congestion level, and energy consumption. By dynamically selecting route paths to adapt to changes in network conditions, congestion can be reduced, data transmission efficiency improved, and latency reduced. Complex routing algorithms require more control logic, which may increase static power consumption. Simple routing algorithms usually reduce this overhead. When designing a NIC, a balance must be found between performance and power consumption. Choosing the right routing algorithm can not only improve data transmission efficiency, but also effectively manage power consumption.

[0005] In router design, the number of virtual channels has a significant impact on latency and power consumption. More virtual channels can effectively reduce network congestion and packet latency, thereby reducing latency. However, more virtual channels mean more control signals and state management, which can increase static power consumption. Switching and managing these channels also leads to increased dynamic power consumption. A balance must be found between latency and power consumption when choosing the number of virtual channels. Too many virtual channels can lead to excessive management overhead, while too few can result in network congestion and increased latency. Using a strategy of dynamically adjusting the number of virtual channels based on actual load conditions can optimize latency and power consumption.

[0006] This invention applies the concept of reinforcement learning to the design of on-chip networks. Reinforcement learning is a field of machine learning that primarily studies how intelligent agents should act in an environment to maximize cumulative rewards. It also continuously adjusts its behavior based on environmental feedback (rewards or penalties) to adapt to the environment and obtain maximum value. By applying reinforcement learning to allocate a suitably sized sub-on-chip network region for the task and selecting an appropriate topology, a high-energy-efficiency on-chip network can be designed.

[0007] This invention employs an adaptive topology to map tasks and divides the network into different regions based on task characteristics for task processing. This method allocates specific on-chip resources and topology types to each task, thereby saving on-chip resource consumption. Through the adaptive topology, the system can flexibly adjust connectivity and routing methods within the on-chip network according to task requirements and characteristics. Simultaneously, by dividing the network into different regions, the processing of different tasks can be isolated, minimizing interference and conflicts. Using Q-learning reinforcement learning algorithms to dynamically change the transmission direction and number of virtual channels during task execution with low latency and low power consumption goals effectively achieves a balance between performance and power consumption in the on-chip network. Summary of the Invention

[0008] To address the challenge of balancing performance and energy consumption in current on-chip network (ISP) data transmission processes, this invention proposes a high-efficiency ISP topology selection method based on reinforcement learning. This method comprises four modules: on-chip resource allocation, topology selection, routing selection, and virtual channel allocation. First, the ISP is divided into different sub-ISP regions, and physical wiring is performed within these regions to allocate appropriate on-chip resources and topologies for different tasks. Second, on-chip resources are allocated to tasks based on their characteristics, and suitable topologies are assigned. Finally, a Q-learning reinforcement learning algorithm is used to select a suitable routing algorithm and allocate an appropriate number of virtual channels during task execution to achieve a balance between performance and power consumption.

[0009] The energy-efficient on-chip network topology selection method based on reinforcement learning includes the following steps:

[0010] Step 1 involves dividing the 16×16 on-chip network into four 8×8 sub-on-chip network regions. These sub-networks employ different topologies for cabling, specifically Torus, Mesh, Tree, and Flattened Butterfly topologies. This division enhances network flexibility and scalability, and lays the foundation for subsequent task mapping and resource allocation.

[0011] Different on-chip network topologies are suitable for different types of tasks. Torus offers high bandwidth and low latency, making it suitable for data-intensive tasks. Mesh provides a simple structure and reasonable performance, making it suitable for compute-intensive tasks. Tree facilitates management and resource allocation, making it suitable for hierarchical tasks and data aggregation. Flattened Butterfly offers good scalability and load balancing capabilities, making it suitable for dynamic loads and high-concurrency tasks. Choosing a suitable topology helps improve task execution efficiency and resource utilization.

[0012] Step 2 involves intelligently mapping tasks to the most suitable sub-on-chip network for processing based on their different characteristics and requirements. This strategy effectively avoids wasting on-chip resources by using the entire on-chip network architecture for a single, simple task. This selective mapping method aims to achieve a balance between high performance and low power consumption, ultimately improving the overall energy efficiency of the on-chip network.

[0013] Step 3: During task execution, based on the size and characteristics of the network on its respective sub-slice, the Q-learning reinforcement learning algorithm is applied to assign the most suitable topology. By learning from historical task performance data, Q-learning can dynamically adjust the network structure to better adapt to the needs of the current task, thereby improving processing efficiency.

[0014] Taking the CPU benchmark suite SPEC2006 as an example, different types of tasks usually have different requirements for topology.

[0015] Data-intensive tasks typically require high-bandwidth and low-latency communication. These tasks are well-suited for Torus or Flattened Butterfly topologies because they offer better parallel communication capabilities and efficient path utilization.

[0016] Computationally intensive tasks have relatively low requirements for network topology. Mesh topologies are relatively simple and can effectively support the needs of such tasks, providing a reasonable balance between performance and power consumption.

[0017] Hierarchical tasks are generally better suited to using a tree topology. This structure facilitates hierarchical management and resource allocation, and can effectively reduce communication latency.

[0018] Dynamic workloads often require flexible topologies to adapt to changing workloads. In this case, the Flattened Butterfly topology provides excellent scalability and load balancing capabilities, and can be adjusted according to real-time requirements.

[0019] Step 4: Using the Q-learning reinforcement learning algorithm, the most suitable routing algorithm and number of virtual channels are selected for the task based on its performance requirements and power consumption constraints. This process not only optimizes the task's execution efficiency but also ensures effective power control while meeting performance requirements, thereby achieving better resource management and system performance. This comprehensive approach effectively improves the energy efficiency of on-chip networks, promoting more efficient computation and data transmission. Q-learning is a model-free, value-based reinforcement learning method suitable for problems with discrete states and action spaces. The system state and possible routing algorithms and the number of virtual channels are abstracted into discrete states and actions. After initializing the Q-table, an ε-greedy policy is used to select actions based on the current state and the Q-table. The routing algorithm and the number of virtual channels are changed according to the selected actions. The Q-values ​​of the corresponding states and actions in the Q-table are updated based on environmental feedback and the Q-learning update rules until convergence or a predetermined number of iterations is reached. In this design, the maximum number of iterations is set to 20,000. Through repeated training and updates, the Q-learning algorithm gradually learns the optimal routing path and the number of virtual channels to maximize the system's performance metrics. Through continuous interaction and learning, reinforcement learning agents can gradually optimize routing algorithms and the number of virtual channels, thereby achieving a balance between performance and power consumption.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] Current on-chip network resources are not fully utilized, resulting in low resource utilization. Furthermore, the lack of targeted task-specific mapping and topology selection schemes for different tasks affects the energy efficiency of on-chip networks. This invention proposes a task-driven on-chip resource allocation and topology selection scheme. It employs a Q-learning reinforcement learning algorithm to allocate appropriately sized sub-on-chip networks based on the characteristics of different tasks, and selects suitable topologies based on the size of the sub-on-chip networks and the characteristics of the tasks.

[0022] On-chip communication may encounter load imbalance issues. Using a uniform number of virtual channels for different tasks can lead to resource waste and insufficient flexibility, as different tasks have varying requirements, and a fixed number of channels may not be able to meet the peak demands of various tasks. Furthermore, this may increase latency and reduce the overall system efficiency. To address this, a dynamic routing algorithm and virtual channel number design scheme based on Q-learning reinforcement learning are introduced. During on-chip resource transmission and computation, the routing algorithm and the number of virtual channels are dynamically adjusted based on real-time latency and power consumption to achieve a balance between power consumption and latency. Attached Figure Description

[0023] To make the objectives and solutions of this invention more easily understood, the invention will be further described below with reference to the accompanying drawings.

[0024] Figure 1 is a framework diagram of a high-efficiency task-driven dynamic on-chip interconnect communication method based on Q-learning;

[0025] Figure 2 shows the routing diagram of a high-efficiency task-driven dynamic on-chip network based on Q-learning;

[0026] Figure 3 shows the topology of Flattened Butterfly.

[0027] Figure 4 shows the working process of Q-learning. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] This invention relates to a high-efficiency task-driven dynamic on-chip interconnect communication method based on Q-learning. As shown in Figure 1, this method consists of four modules: on-chip resource allocation, topology selection, routing selection, and virtual channel allocation. First, the 16×16 on-chip network is divided into four 8×8 sub-on-chip network regions. Within these sub-on-chip networks, routing is performed according to Torus, Mesh, Tree, and Flattened Butterfly topologies, respectively. Then, based on the different characteristics of the task, it is mapped to an appropriate sub-on-chip network for processing. This method effectively avoids wasting on-chip resources by using the entire on-chip network architecture for a simple task, balancing high performance and low power consumption, and improving the energy efficiency of the on-chip network. Finally, on the sub-on-chip networks, based on the size of the sub-on-chip network where the task is running and the characteristics of the task, a suitable topology is allocated using a Q-learning reinforcement learning algorithm. Furthermore, Q-learning reinforcement learning is used to select an appropriate routing algorithm and the number of virtual channels for the task based on performance and power consumption, achieving a balance between performance and power consumption. The specific steps are as follows:

[0030] Step 1: Divide the 16×16 on-chip network into four 8×8 sub-on-chip network regions. These sub-networks are routed using different topologies, including Torus, Mesh, Tree, and Flattened Butterfly topologies. The specific on-chip network routing diagram is shown in Figure 2. However, due to the complexity of the Flattened Butterfly topology, the Flattened Butterfly topology portion in Figure 2 is a conceptual diagram, as shown in Figure 3.

[0031] Step 2 uses the SPEC CPU 2006 benchmark as the task source. Tasks in SPEC CPU 2006 are categorized into compute-intensive tasks, data-intensive tasks, hierarchical tasks, and high-concurrency tasks. Torus offers high bandwidth and low latency, suitable for data-intensive tasks. Mesh provides a simple structure and reasonable performance, suitable for compute-intensive tasks. Tree facilitates management and resource allocation, suitable for hierarchical tasks and data aggregation. Flattened Butterfly has good scalability and load balancing capabilities, suitable for dynamic loads and high-concurrency tasks. Choosing a suitable topology helps improve task execution efficiency and resource utilization.

[0032] Step 3: Based on the different characteristics and requirements of the tasks, the Q-learning reinforcement learning algorithm is used to intelligently map them to the most suitable sub-on-chip networks for processing. This strategy effectively avoids wasting on-chip resources by using the entire on-chip network architecture for a single simple task. The parallel execution capability of each task is evaluated using the Q-learning reinforcement learning algorithm. Some tasks can be divided into multiple threads, while others may have serial dependencies. The number of CPU cores is dynamically allocated according to the parallel capability of the task, so that the task is evenly distributed across all cores to achieve load balancing. This selective mapping method aims to achieve a balance between high performance and low power consumption, ultimately improving the overall energy efficiency of the on-chip network. During task execution, the Q-learning reinforcement learning algorithm is applied to assign the most suitable topology to the sub-on-chip network based on its size and characteristics. By learning from the performance data of historical tasks, Q-learning can dynamically adjust the network structure to better adapt to the needs of the current task, thereby improving processing efficiency. In this design, the Q-learning reinforcement learning agent uses a task-mapped on-chip resource allocation and topology selection strategy. The agent selects a sub-on-chip network region for the current task based on the required topology and communicates data according to the routing within that topology; this process is the action. The power consumption and performance generated during this process constitute the environment. The environment sends rewards or penalties to the agent: a reward for improved performance and a penalty for decreased performance; a reward for decreased power consumption and a penalty for decreased power consumption. The environment also updates the agent's current state, and the agent takes the next action based on the reward or penalty.

[0033] The goal of this design is to find a balance between performance and power consumption in dynamic on-chip networks driven by high-energy-efficiency tasks. The performance metric chosen is latency. If the reinforcement learning agent continuously searches for the topology with the lowest latency, and then searches for the topology with the lowest power consumption, a deadlock will occur. This is because the latency and power consumption of an on-chip network topology cannot be optimal simultaneously; the Flattened Butterfly topology, with the lowest latency, has the highest power consumption, while the Mesh topology, with the lowest power consumption, has the highest latency. Therefore, to select the most suitable topology for different tasks, both performance and power consumption must be considered comprehensively. This design uses the power-latency product, PDP, which is the product of power and latency, to express the balance between power and latency. PDP = P avg *T d

[0034] The specific Q-learning task allocation and topology selection algorithm is as follows.

[0035] 1) Initialize the Q-table: Initialize the Q-value for each state and action combination, setting all Q-values ​​in the table to 0. The state space represents the delay-power product and on-chip resource utilization of the on-chip network, while the action space represents the regions and topologies of the sub-on-chip network.

[0036] 2) Action selection: An ε-greedy strategy is used to balance exploration and exploitation. Most of the time, the action with the highest current Q value is selected, but there is a small probability of random selection to explore new strategies.

[0037] 3) Execute action: Adjust the allocation of resource areas on the task chip according to the selected action.

[0038] 4) Obtain feedback: Observe the system's latency and power consumption after execution.

[0039] 5) Update the Q-value: Based on environmental feedback and the Q-learning update formula: In the above equation, Q(s,a) on the left-hand side represents the state. Next action The new Q value, Q(s,a) on the right is the original Q value, r is the reward obtained after performing action a, and max a′ Q(s′,a′) is the next state. Below, all actions The maximum Q value is given by α, s′ is the next state, α is the learning rate, and γ is the discount factor, with parameters 0≤α≤1 and 0≤γ≤1. In this design, the learning rate α is set to 0.01, the discount factor γ is set to 0.9, and the greedy factor ε is set to 0.2.

[0040] Step 4: Use the Q-learning reinforcement learning algorithm for route selection and setting the number of virtual channels. The working process of the reinforcement learning algorithm is shown in Figure 4. First, define the state space and action space.

[0041] State space: Defines the state in the network, such as the load of each node, link status, current route, etc.

[0042] Action space: Defines the number of selectable routing paths and virtual channels.

[0043] This invention sets the number of virtual channels to 1 to 6. Available routing algorithms include XY routing, adaptive routing, and weighted routing. XY routing is a deterministic routing algorithm commonly used in Mesh or Torus topologies. In this algorithm, packets first move along the X-axis until they reach the column of the destination node, then move along the Y-axis until they reach the destination node. The advantage of this routing method is its simplicity and directness, but the disadvantage is that it can potentially cause communication bottlenecks in some parts of the network. Adaptive routing can dynamically change the routing path based on network conditions at runtime. In weighted routing, each possible path or link is assigned a weight value. Routing decisions are based on these weight values, and packets tend to choose the path with the highest or lowest weight. Weights can be assigned based on many factors, such as path length, congestion, or energy consumption. This method can provide better performance than XY routing, but it is more complex to design and implement and may require more hardware resources.

[0044] In this design, the reinforcement learning agent uses an online routing algorithm based on Q-learning and a virtual channel number selection strategy. The agent communicates based on the current routing algorithm and the number of virtual channels; this process is the action. The on-chip network's action space consists of the three routing algorithms and the number of virtual channels, while the on-chip network's delay-power product and link load state constitute the state space.

[0045] The training process is as follows.

[0046] 1) Initialize a Q-value table Q(s,a), where each state... and actions The Q value is initialized to zero.

[0047] 2) Based on the current state Choose an action from the Q-value table.

[0048] 3) Use the ε-greedy strategy, randomly select an action with probability ε for exploration, and select the action with the largest Q value with probability 1-ε for exploitation.

[0049] 4) Based on the new state after the action is performed. Calculate the reward r.

[0050] 5) Update the Q-value: Based on environmental feedback and the Q-learning update formula: In the above equation, Q(s,a) on the left-hand side represents the state. Next action The new Q value, Q(s,a) on the right is the original Q value, r is the reward obtained after performing action a, and max a′ Q(s′,a′) is the next state. Below, all actions The maximum Q value is given by α, s′ is the next state, α is the learning rate, and γ is the discount factor, with parameters 0≤α≤1 and 0≤γ≤1. In this design, the learning rate α is set to 0.01, the discount factor γ is set to 0.9, and the greedy factor ε is set to 0.2.

[0051] The Q-learning reinforcement learning algorithm is used to dynamically change the transmission direction of the routing algorithm and the number of virtual channels during task execution with the goal of low latency and low power consumption, so as to achieve a balance between performance and power consumption in the on-chip network.

Claims

1. A high-efficiency task-driven dynamic on-chip interconnect communication method based on Q-learning, characterized in that, Includes the following steps: Step 1: Divide the 16×16 on-chip network into four 8×8 sub-on-chip network regions; these sub-networks are wired using different topologies, specifically including Torus topology, Mesh topology, Tree topology and Flattened Butterfly topology. Step 2: Based on the different characteristics and requirements of the task, map it to the appropriate sub-slice network; Torus is suitable for data-intensive tasks; Mesh is suitable for computationally intensive tasks; Tree is suitable for hierarchical tasks and data aggregation; Flattened Butterfly is suitable for dynamic loads and high-concurrency tasks. Step 3: During task execution, based on the size and characteristics of the network on the sub-slice, apply the Q-learning reinforcement learning algorithm to assign a suitable topology to it. Step 4: Using the Q-learning reinforcement learning algorithm, select the routing algorithm and the number of virtual channels for the task based on the task's performance requirements and power consumption constraints; The system state, possible routing algorithms, and number of virtual channels are abstracted into discrete states and actions. After initializing the Q-table, an ε-greedy strategy is used to select an action based on the current state and the Q-table. Based on the selected action, the routing algorithm and the number of virtual channels are changed. Based on the feedback from the environment and the update rules of Q-learning, the Q-values ​​of the corresponding states and actions in the Q-table are updated until convergence or a predetermined number of iterations is reached. The maximum number of iterations is set to 20,000.

Images