Q-learning-based high-energy-efficiency task-driven dynamic on-chip interconnection communication method

Abstract

The invention discloses a Q-learning-based high-energy-efficiency task-driven dynamic on-chip interconnection communication method, and belongs to the field of on-chip interconnection of computer systems. According to the method, a self-adaptive topological structure is adopted to map tasks, and different areas are divided according to different features of the tasks for task processing. According to the method, specific on-chip resources and topological structure types can be allocated to each task, so that the consumption of the on-chip resources is reduced. Through the self-adaptive topological structure, the system can flexibly adjust the connection relation and the routing mode in the network-on-chip according to the requirements and characteristics of tasks. Meanwhile, by dividing different areas, the processing processes of different tasks can be isolated, and interference and conflicts are reduced to the maximum extent. The Q-learning reinforcement learning algorithm is used for dynamically changing the transmission direction of the routing algorithm and the number of virtual channels by taking low delay and low power consumption as targets during task operation, so that the balance between the performance and the power consumption in the network-on-chip can be effectively realized.

Description

Technical Field

[0001] The present invention belongs to the field of on-chip interconnection of computer systems, and in particular relates to an adaptive on-chip network solution based on Q-learning reinforcement learning. Background Art

[0002] Modern processors usually have multiple cores. On-chip networks play an important role in chip multiprocessors and multiprocessor systems-on-chip. They solve the problem of communication between processing cores in multi-core processors and provide low-latency, high-bandwidth, and scalable communication channels. In current research, the resources of on-chip networks are usually not fully utilized, and there is a problem of low resource utilization. In the task mapping process, it is necessary to consider the load balancing between different core resources to avoid performance loss caused by idle core resources. At the same time, when processing different tasks, on-chip networks lack targeted specific task topology selection schemes. The topology of the on-chip network determines the physical layout and connection between router nodes and transmission channels in the network, which directly affects the routing selection of messages and directly affects network latency. The number of virtual channels in the router also has a significant impact on latency and power consumption.

[0003] The network on chip is composed of router nodes and transmission channels. The choice of topology has an important impact on the transmission performance and power consumption of the network on chip. The topology determines the number of hops or the number of routers a message passes through, so it will affect the network delay. Messages need to consume energy when passing through routers and links, so the impact of the topology on the number of hops will directly affect the power consumption of the system. The topology is a very important part of the network on chip design. It has a very large impact on the overall performance and overhead of the network on chip. The topology determines the physical distribution of the network nodes and links on the chip and their connection relationship. At the same time, it also determines the number of message transmission hops. Therefore, the design of the topology has a great impact on the transmission delay and power consumption. At present, there are three direct connection topologies in the network on chip: ring network, two-dimensional mesh network and two-dimensional Torus network. Non-direct connection topologies include butterfly and tree structures. Different topologies are suitable for different types of tasks to transmit data. In previous research on on-chip network interconnection structures, all tasks often use a single topology connection method. This results in additional overhead for some tasks because such complex wiring is not required, wasting power consumption. For other tasks, too many hops in message transmission increase latency and reduce performance.

[0004] The topology of the on-chip network directly affects the choice of routing algorithms. Routing algorithms in on-chip networks are generally divided into three categories: deterministic routing, oblivious routing, and adaptive routing. Deterministic routing is a basic routing method. For a given source and destination, the routing path is fixed, such as XY routing. The path in oblivious routing is determined by the source and destination nodes, but does not take into account the real-time or historical congestion information of the network. Therefore, even if some paths are congested, the routing selection will not change. Adaptive routing is a more complex routing strategy that can dynamically change the routing path according to network conditions at runtime. In weight-based routing, each path or link has a weight, and the routing selection depends on these weights. The weight can be set based on a variety of factors, such as path length, congestion level, energy consumption, etc. By dynamically selecting routing paths and adapting to changes in network status, congestion can be reduced, data transmission efficiency can be improved, and latency can be reduced. Complex routing algorithms require more control logic, which may increase static power consumption. Simple routing algorithms can usually reduce this overhead. When designing on-chip networks, it is necessary to find a balance between performance and power consumption. Choosing the right routing algorithm can not only improve data transmission efficiency but also effectively manage power consumption.

[0005] In router design, the number of virtual channels has a significant impact on latency and power consumption. More virtual channels can effectively reduce network congestion and packet latency, thereby reducing latency. However, more virtual channels mean that more control signals and state management are required, which may increase static power consumption. Switching and managing these channels will also cause dynamic power consumption to rise. When choosing the number of virtual channels, it is necessary to find a balance between latency and power consumption. Too many virtual channels may cause excessive management overhead, while too few may cause network congestion and increased latency. Using a strategy to dynamically adjust the number of virtual channels and flexibly adjust according to actual load conditions can optimize latency and power consumption.

[0006] The present invention applies the idea of ​​reinforcement learning to the design of on-chip networks. Reinforcement learning is a field of machine learning that mainly studies how intelligent agents should take actions in the environment to maximize the cumulative rewards they obtain. They will also continuously adjust their behaviors based on the feedback of the environment on the behavior, that is, rewards or punishments, so as to adapt to the environment and obtain the maximum value. Reinforcement learning is used to allocate a sub-chip network area of ​​a suitable size for the task and select a suitable topology structure to design an energy-efficient on-chip network.

[0007] The present invention uses an adaptive topology structure to map tasks, and divides different areas for task processing according to the characteristics of the tasks. This method can allocate specific on-chip resources and topology types to each task, thereby saving on-chip resource consumption. Through the adaptive topology structure, the system can flexibly adjust the connection relationship and routing method in the on-chip network according to the needs and characteristics of the task. At the same time, by dividing different areas, the processing processes of different tasks can be isolated, minimizing the occurrence of interference and conflicts. Using the Q-learning reinforcement learning algorithm to dynamically change the transmission direction of the routing algorithm and the number of virtual channels with the goal of low latency and low power consumption when the task is running can effectively achieve a balance between performance and power consumption in the on-chip network. Summary of the invention

[0008] In order to solve the problem that it is difficult to achieve a balance between performance and energy consumption in the current on-chip network data transmission process, the present invention proposes a high-efficiency on-chip network topology selection method based on reinforcement learning. This topology selection method consists of four modules, namely on-chip resource allocation, topology selection, routing selection, and virtual channel allocation. First, the on-chip network needs to be divided into different sub-on-chip network areas, and physical wiring is performed in the sub-on-chip network areas so that appropriate on-chip resources and topologies can be allocated to different tasks; secondly, on-chip resources are allocated to tasks according to different characteristics and appropriate topologies are allocated; finally, the Q-learning reinforcement learning algorithm is used to select the appropriate routing algorithm and allocate the appropriate number of virtual channels when the task is running to achieve a balance between performance and power consumption.

[0009] The energy-efficient on-chip network topology selection method based on reinforcement learning includes the following steps:

[0010] Step 1: Divide the 16×16 on-chip network into four 8×8 sub-on-chip network areas. These sub-networks are wired using different topologies, including Torus topology, Mesh topology, Tree topology, and Flattened Butterfly topology. This division improves the flexibility and scalability of the network and lays the foundation for subsequent task mapping and resource allocation.

[0011] Different on-chip network topologies are suitable for different types of tasks. Torus provides high bandwidth and low latency, suitable for data-intensive tasks. Mesh provides a simple structure and reasonable performance, suitable for compute-intensive tasks. Tree is easy to manage and allocate resources, suitable for hierarchical tasks and data aggregation. Flattened Butterfly has good scalability and load balancing capabilities, suitable for dynamic loads and high-concurrency tasks. Choosing a suitable topology helps improve task execution efficiency and resource utilization.

[0012] Step 2: Intelligently map the task to the most appropriate sub-on-chip network for processing based on its different characteristics and requirements. This strategy can effectively avoid using the entire on-chip network architecture for a single simple task, which would lead to a waste of on-chip resources. This selective mapping method aims to achieve a balance between high performance and low power consumption, ultimately improving the overall energy efficiency of the on-chip network.

[0013] Step 3: When the task is running, the Q-learning reinforcement learning algorithm is applied to assign the most suitable topology structure to it according to the scale and characteristics of the network on the sub-chip where it is located. By learning the performance data of historical tasks, Q-learning can dynamically adjust the network structure to better adapt to the needs of the current task, thereby improving processing efficiency.

[0014] Taking the CPU benchmark suite SPEC2006 as an example, different types of tasks usually have different requirements for topology structures.

[0015] Data-intensive tasks usually require high-bandwidth and low-latency communication. Such tasks are suitable for using Torus or Flattened Butterfly topologies because they provide better parallel communication capabilities and efficient path utilization.

[0016] Computation-intensive tasks have relatively low requirements on the network topology. The Mesh topology is relatively simple and can effectively support the needs of such tasks, providing a reasonable balance between performance and power consumption.

[0017] Hierarchical tasks are usually more suitable for using the Tree topology. This structure is conducive to hierarchical management and resource allocation, and can effectively reduce communication delays.

[0018] Dynamic load tasks often require flexible topologies to adapt to changing task loads. In this case, the Flattened Butterfly topology provides good scalability and load balancing capabilities and can be adjusted according to real-time needs.

[0019] Step 4: Use the Q-learning reinforcement learning algorithm to select the most appropriate routing algorithm and number of virtual channels for the task based on the performance requirements and power consumption constraints of the task. This process not only optimizes the execution efficiency of the task, but also ensures that power consumption is effectively controlled while meeting performance requirements, thereby achieving better resource management and system performance. This comprehensive approach can effectively improve the energy efficiency of the on-chip network and promote more efficient computing and data transmission. Q-learning is a model-free, value-based reinforcement learning method that is suitable for problems with discrete states and action spaces. The system state and the possible routing algorithms and number of virtual channels are abstracted into discrete states and actions. After initializing the Q table, the ε-greedy strategy is used to select actions based on the current state and Q table. According to the selected action, the routing algorithm and the number of virtual channels are changed. According to the feedback from the environment and the update rules of Q-learning, the Q value of the corresponding state and action in the Q table is updated until convergence or the predetermined number of iterations is reached. In this design, the maximum number of iterations is set to 20,000. Through repeated training and updating, the Q-learning algorithm can gradually learn the optimal routing transmission route and number of virtual channels to maximize the performance indicators of the system. Through continuous interaction and learning, the reinforcement learning agent can gradually optimize the routing algorithm and the number of virtual channels to meet the balance between performance and power consumption.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] The resources of current on-chip networks are not fully utilized, and there are problems such as low resource utilization. At the same time, when processing different tasks, the on-chip network lacks targeted specific task mapping solutions and topology selection solutions, which will affect the energy efficiency of the on-chip network. The present invention proposes a task-driven on-chip resource allocation and topology selection solution, which uses the Q-learning reinforcement learning algorithm to allocate a sub-on-chip network of appropriate size according to the characteristics of the task, and selects a suitable topology for it according to the size of the sub-on-chip network and the characteristics of the task.

[0022] Load imbalance may occur during on-chip communication. Using a unified number of virtual channels for different tasks may lead to waste of resources and lack of flexibility, because the requirements of different tasks vary, and a fixed number of channels may not be able to adapt to the peak requirements of various tasks. In addition, this may increase latency and reduce the overall efficiency of the system. To this end, a dynamic routing algorithm and virtual channel number design based on Q-learning reinforcement learning are introduced. During on-chip resource transmission and calculation, the routing algorithm and the number of virtual channels are dynamically changed according to the real-time latency and power consumption to achieve a balance between power consumption and latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to make the purpose and scheme of the present invention more understandable, the present invention will be further described below in conjunction with the accompanying drawings.

[0024] Figure 1 A framework diagram of a dynamic on-chip interconnect communication method for energy-efficient task driving based on Q-learning;

[0025] Figure 2 Driving dynamic on-chip network wiring diagrams for energy-efficient tasks based on Q-learning;

[0026] Figure 3 This is the topology diagram of FlattenedButterfly;

[0027] Figure 4 Diagram of the Q-learning working process. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] The present invention relates to a high-energy-efficiency task-driven dynamic on-chip interconnect communication method based on Q-learning, such as Figure 1 As shown in the figure, the energy-efficient task-driven dynamic on-chip interconnect communication method based on Q-learning consists of four modules, namely on-chip resource allocation, topology selection, routing selection, and virtual channel allocation. First, the 16×16 on-chip network is divided into four 8×8 sub-on-chip network areas. In the sub-on-chip network, the wiring is performed according to the Torus topology, Mesh topology, Tree topology, and Flattened Butterfly topology. Then, according to the different characteristics of the task, it is mapped to the appropriate sub-on-chip network for processing. This method can effectively avoid the waste of on-chip resources caused by using the entire on-chip network architecture for a simple task, balance the goals of high performance and low power consumption, and improve the energy efficiency of the on-chip network. Finally, on the sub-on-chip network, according to the scale of the sub-on-chip network where the task is running and the characteristics of the task, the Q-learning reinforcement learning algorithm is used to allocate a suitable topology for it, and the Q-learning reinforcement learning is used to select the appropriate routing algorithm and number of virtual channels for the task according to performance and power consumption to meet the balance of performance and power consumption. The specific steps are as follows:

[0030] Step 1: Divide the 16×16 on-chip network into four 8×8 sub-on-chip network areas. These sub-networks are wired using different topologies, including Torus topology, Mesh topology, Tree topology, and Flattened Butterfly topology. The specific on-chip network wiring diagram is as follows: Figure 2 As shown in the figure, due to the complex topological structure of FlattenedButterfly, Figure 2 The Flattened Butterfly topology is a conceptual diagram. Figure 3 shown.

[0031] Step 2, using the SPEC CPU 2006 benchmark as the task source. The tasks in SPEC CPU 2006 are divided into compute-intensive tasks, data-intensive tasks, hierarchical tasks, and high-concurrency tasks. Torus provides high bandwidth and low latency, suitable for data-intensive tasks. Mesh provides a simple structure and reasonable performance, suitable for compute-intensive tasks. Tree is easy to manage and allocate resources, suitable for hierarchical tasks and data aggregation. Flattened Butterfly has good scalability and load balancing capabilities, suitable for dynamic loads and high-concurrency tasks. Choosing a suitable topology structure helps improve task execution efficiency and resource utilization.

[0032] Step 3: Based on the different characteristics and requirements of the tasks, the Q-learning reinforcement learning algorithm is used to intelligently map them to the most appropriate sub-on-chip network for processing. This strategy can effectively avoid using the entire on-chip network architecture for a single simple task, which leads to a waste of on-chip resources. The Q-learning reinforcement learning algorithm is used to evaluate the parallel execution capability of each task. Some tasks can be divided into multiple threads, while some tasks may have serial dependencies. The number of CPU cores is dynamically allocated to the tasks according to their parallel capabilities, so that the tasks are evenly distributed to all cores to achieve load balancing. This selective mapping method aims to achieve a balance between high performance and low power consumption, and ultimately improve the overall energy efficiency of the on-chip network. When the task is running, the Q-learning reinforcement learning algorithm is applied to allocate the most suitable topology structure to it based on the scale and characteristics of the sub-on-chip network it is in. By learning the performance data of historical tasks, Q-learning can dynamically adjust the network structure to better adapt to the needs of the current task, thereby improving processing efficiency. In this design, the intelligent agent of Q-learning reinforcement learning is based on the on-chip resource allocation and topology selection strategy of task mapping. The intelligent agent selects the sub-chip network area for the current task based on the topology required by the current task, and communicates data according to the route under the topology. This process is behavior. The power consumption and performance generated by this process are the environment. The environment sends rewards or penalties to the intelligent agent. If the performance is improved, it will be rewarded, otherwise it will be punished; if the power consumption is reduced, it will be rewarded, otherwise it will be punished, and the current state of the intelligent agent will be updated. The intelligent agent will take the next action based on the reward or punishment.

[0033] The purpose of this design is to find a balance between performance and power consumption in a dynamic on-chip network driven by energy-efficient tasks. Latency is selected as the performance indicator. If the reinforcement learning agent is always looking for the topology with the lowest latency, and then looking for the topology with the lowest power consumption, a deadlock will occur. Because the latency and power consumption of the on-chip network topology cannot be optimized at the same time, the FlattenedButterfly with the lowest latency has the highest power consumption, while the Mesh with the lowest power consumption has the highest latency. Therefore, if you want to choose the most suitable topology for different tasks, you must consider performance and power consumption comprehensively and link performance and power consumption. This design uses the power delay product, that is, the product of power consumption and delay latency, PDP, to express the balance between power consumption and delay.

[0034] PDP=P avg *T d

[0035] The specific Q-learning task allocation and topology selection algorithm is as follows.

[0036] 1) Initialize the Q table: Initialize the Q value for each state and action combination, and set all Q values ​​in the table to 0. The state space is the product of the delay power consumption of the on-chip network and the on-chip resource utilization, and the action space is the area of ​​the sub-on-chip network and the topological structure therein.

[0037] 2) Action selection: Use the ε-greedy strategy to balance exploration and exploitation. Most of the time, the action with the highest current Q value is selected, but there is also a small probability of random selection to explore new strategies.

[0038] 3) Execute action: adjust the allocation of resource areas on the task slice according to the selected action.

[0039] 4) Get feedback: Observe the latency and power consumption of the system after execution.

[0040] 5) Update Q value: Based on the environment feedback and Q-learning update formula: Among them, Q(s, a) on the left side of the above formula is the new Q value of action a in state s, Q(s, a) on the right side is the original Q value, r is the reward obtained after executing action a, and max a′ Q(s′, a′) is the maximum Q value of all actions a′ in the next state s′, s′ is the next state, α is the learning rate, γ is the discount factor, and the parameters are 0≤α≤1, 0≤Υ≤1. In this design, the learning rate α is set to 0.01, the discount factor γ is set to 0.9, and the greed factor ε is set to 0.2.

[0041] Step 4: Use the Q-learning reinforcement learning algorithm to select routes and set the number of virtual channels. The working process of the reinforcement learning algorithm is as follows: Figure 4 As shown. First, define the state space and action space.

[0042] State space: defines the state in the network, such as the load of each node, link status, current routing, etc.

[0043] Action space: defines the number of selectable routing paths and virtual channels.

[0044] The present invention sets the number of virtual channels to 1 to 6. The routing algorithms available are XY routing, adaptive routing, and weight-based routing. XY routing is a deterministic routing algorithm commonly used in Mesh or Torus topologies. In this algorithm, a data packet first moves on the X-axis until it reaches the column of the target node, and then moves on the Y-axis until it reaches the target node. The advantages of this routing method are simplicity and directness, and the disadvantage is that it may cause communication bottlenecks in some parts of the network. Adaptive routing can dynamically change the routing path according to network conditions at runtime. In weight-based routing, each possible path or link is assigned a weight value. Routing decisions are based on these weight values, and data packets tend to choose the path with the highest or lowest weight. Weights can be assigned based on many factors, such as path length, congestion conditions, or energy consumption. This method can provide better performance than XY routing, but the design and implementation are more complex and may require more hardware resources.

[0045] In this design, the reinforcement learning agent is based on the online routing algorithm and virtual channel number selection strategy of Q-learning. The agent communicates based on the current routing algorithm and the number of virtual channels. This process is behavior. The action space of the on-chip network is the three routing algorithms and the number of virtual channels. The delay power consumption product of the on-chip network and the load state of the link are the state space.

[0046] The training process is as follows.

[0047] 1) Initialize a Q value table Each of these states And the Q-value of action a is initialized to zero.

[0048] 2) According to the current status And the Q-value table selects an action a.

[0049] 3) Using the ε-greedy strategy, randomly select an action as exploration with probability ε, and select the action with the largest Q value as exploitation with probability 1-ε.

[0050] 4) Based on the new state after executing the action Calculate the reward r.

[0051] 5) Update Q value: Based on the environment feedback and Q-learning update formula: Among them, Q(s, a) on the left side of the above formula is the new Q value of action a in state s, Q(s, a) on the right side is the original Q value, r is the reward obtained after executing action a, and max a′Q(s′, a′) is the maximum Q value of all actions a′ in the next state s′, s′ is the next state, α is the learning rate, γ is the discount factor, and the parameters are 0≤α≤1, 0≤Υ≤1. In this design, the learning rate α is set to 0.01, the discount factor γ is set to 0.9, and the greed factor ε is set to 0.2.

[0052] The Q-learning reinforcement learning algorithm is used to dynamically change the transmission direction and the number of virtual channels of the routing algorithm with the goal of low latency and low power consumption during task execution to achieve a balance between performance and power consumption in on-chip networks.

Claims

1. A high-energy-efficiency task-driven dynamic on-chip interconnect communication method based on Q-learning, characterized in that: The following steps are involved: Step 1: Divide the 16×16 on-chip network into four 8×8 sub-on-chip network areas; these sub-networks are wired using different topologies, including Torus topology, Mesh topology, Tree topology, and Flattened Butterfly topology; Step 2: Map the tasks to appropriate sub-on-chip networks according to their different characteristics and requirements; Torus is suitable for data-intensive tasks; Mesh is suitable for computation-intensive tasks; Tree is suitable for hierarchical tasks and data aggregation; Flattened Butterfly is suitable for dynamic loads and high-concurrency tasks; Step 3: When the task is running, the Q-learning reinforcement learning algorithm is applied to assign a suitable topology structure to the task according to the size and characteristics of the network on the sub-chip where the task is located; Step 4: Use the Q-learning reinforcement learning algorithm to select the routing algorithm and number of virtual channels for the task based on the performance requirements and power consumption constraints of the task; The system state and possible routing algorithms and the number of virtual channels are abstracted into discrete states and actions. After initializing the Q-table, the ε-greedy strategy is used to select actions according to the current state and Q-table. The routing algorithm and the number of virtual channels are changed according to the selected action. The Q values ​​of the corresponding states and actions in the Q-table are updated according to the feedback from the environment and the update rules of Q-learning until convergence or the predetermined number of iterations is reached. The maximum number of iterations is set to 20,000.

Images