A synchronization-aware arbitration module for a network-on-chip based neural network accelerator

Abstract

This invention discloses a synchronization-aware arbitration module for network-on-chip (NAT) based neural network accelerators. It includes a priority generation module integrated in a processing unit, which receives data packets, sends valid signals, assigns priorities to data packets based on the number of remaining data packets in the processing unit, and generates corresponding layer identifiers. The synchronization-aware arbitration module, integrated in a router, receives the priority and layer identifier carried in data packets and performs synchronization-aware arbitration for inter-layer communication. This invention addresses the communication synchronization problem caused by many-to-one and many-to-many traffic in NAT-based neural network accelerators. It achieves backpressure control of source nodes through synchronization-aware priority arbitration of adjacent layer data packets, dynamically adjusts the packet sending rate of each source node through a negative feedback mechanism, realizes communication synchronization between layers, reduces transmission delay between layers, and ultimately reduces the total inference time of the neural network.

Description

Technical Field

[0001] This invention relates to the field of on-chip network technology and discloses a synchronous sensing arbitration module for on-chip network-based neural network accelerators. Background Technology

[0002] In recent years, neural networks (NNs) have been widely used in various fields, playing a crucial role in image recognition and semantic analysis. Currently, many neural network accelerators have been proposed. Besides the computational power of each processing element (PE), the frequent data interaction between processing elements is also a significant factor affecting accelerator performance. To improve communication bandwidth and scalability, many accelerators use on-chip networks instead of traditional bus architectures for communication.

[0003] Routers are a crucial component of Network-on-Chip (NoC), and the arbitration strategy employed by routers is a significant factor influencing NoC performance while ensuring network fairness. Common arbitration policies include Round-Robin (RR), First-In-First-Out (FIFO), and Global-Age (GA). These traditional arbitration strategies typically consider the NoC in isolation, neglecting real-world traffic patterns and optimizations. While they may yield good results in some synthetic traffic tests, they can perform poorly in practical neural network applications. In NoC-based neural network accelerators, traffic exists between different layers' Process Entities (PEs), often exhibiting many-to-one and many-to-many traffic patterns. Each PE needs to receive input from multiple PEs in the preceding layer, and can only begin computation after receiving all the input neurons required for the current round of computation, leading to synchronization issues between different PEs. Summary of the Invention

[0004] Purpose of the invention: To provide a synchronous sensing arbitration module for on-chip network-based neural network accelerators, in order to solve the above-mentioned problems in the prior art, namely the communication synchronization problem caused by many-to-one and many-to-many traffic.

[0005] Technical solution: A synchronous sensing arbitration module for on-chip network-based neural network accelerators, comprising:

[0006] The priority generation module, integrated into the processing unit, assigns priorities to data packets and generates corresponding layer identifiers;

[0007] The synchronous sensing arbitration module, integrated in the router, receives the priority and layer identifier carried in the data packets and performs synchronous sensing arbitration for inter-layer communication.

[0008] According to one aspect of this application, the priority allocation for data packets is specifically as follows:

[0009] The system receives valid signals from data packets, calculates the scaling factor for each processing unit, and generates priorities. The formula for the scaling factor C of each processing unit is as follows:

[0010]

[0011] Where C is the scaling factor for each processing unit, and N... total The total number of packets generated per inference process for each processing unit, where B is the bit width used for priority, and the priority formula is as follows:

[0012]

[0013] in For the floor function, N trans and N left These represent the number of data packets already sent and the number of remaining data packets for each processing unit, respectively.

[0014] According to one aspect of this application, the priority generation module includes:

[0015] The layer identifier register stores the neural network layer number that the processing unit is responsible for. It is assigned a value during the configuration phase of each processing unit. The layer identifier of the data packet is given by this register.

[0016] The scaling counter generates an output to the priority counter every C data packets received, sending a valid signal. This is used to scale the number of data packets sent, corresponding to the value in the Priority expression.

[0017] The priority counter decrements by 1 each time it receives the output of the scaling counter. This count value represents the priority. The priority generation module works as follows:

[0018] Sa, the value L of the configuration layer identifier register, the initial value C of the scaling counter, and the initial value N of the priority counter, where N corresponds to the value in the Priority expression.

[0019] When sending a data packet, the scaling counter is decremented by 1. When the scaling counter reaches 0, the initial value C of the scaling counter is reset. At the same time, the priority counter is decremented by 1. The current count value of the counter is output, which is the priority of the data packet. The value of the layer identifier register is read, which is the layer identifier of the data packet.

[0020] Sc, When the priority counter reaches 0, reset the initial value N of the priority counter.

[0021] According to one aspect of this application, the synchronous sensing arbitration module includes at least five sub-arbiters, each sub-arbiter corresponding to an output direction, performing inter-layer communication synchronous sensing arbitration on all data packets requesting that output direction, and outputting successfully arbitrated data packets according to the priority and layer identifier in the data packets.

[0022] According to one aspect of this application, the specific steps of inter-layer communication synchronous sensing arbitration are as follows:

[0023] Priority and layer identifier in received data packets;

[0024] For data packets originating from the same layer, select the data packet with the highest priority.

[0025] For data packets originating from different layers and data packets with the same priority within the same layer, a round-robin arbitration algorithm is used to give these data packets the same priority.

[0026] According to one aspect of this application, it also includes using a negative feedback mechanism to adjust the number of remaining data packets in different processing units at the same layer, wherein the negative feedback mechanism specifically includes:

[0027] The formula for data packet transmission delay is:

[0028] T = Tqueue + ∑1 hops (Tpipeline+Tblock+Tcontention+Tlink)+Tsize;

[0029] Where Tqueue is the packet queuing delay, Tsize is the delay determined by the packet size, Tpipeline is the delay caused by the number of router pipeline stages, Tlink is the link transmission delay, Tblock is the delay caused by downstream router congestion, and Tcontention is the delay caused by packet contention.

[0030] Compared with other processing units in the same layer, it has more remaining data packets, higher priority, reduced Tcontention, reduced transmission delay, and fewer remaining data packets;

[0031] Compared to other processing units in the same layer, there are fewer remaining data packets, lower priority, increased Tcontention, increased transmission delay, and more remaining data packets.

[0032] Beneficial effects: This invention optimizes the communication synchronization problem in neural network accelerators based on on-chip networks. By assigning priority to each data packet and dynamically adjusting the packet sending rate of the source node, inter-layer communication synchronization is achieved, inter-layer transmission delay is reduced, and the overall neural network inference time is reduced, which has good practical application value. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of a neural network mapped to the NoC platform.

[0034] Figure 2 This diagram illustrates the communication synchronization problem on a neural network accelerator based on on-chip networks.

[0035] Figure 3 This is a schematic diagram of the back pressure and negative feedback mechanism.

[0036] Figure 4 Generate a module logic diagram to indicate the priority of the processing unit.

[0037] Figure 5 This is a diagram illustrating the router arbitration process.

[0038] Figure 6 This diagram illustrates the inference time for different arbitration strategies under different neural networks. Detailed Implementation

[0039] Neural networks have been widely used in many fields, playing an important role in image recognition, semantic analysis, etc. This application proposes a synchronous perception arbitration module for neural network accelerators based on on-chip networks, specifically including:

[0040] The priority generation module, integrated into the processing unit, assigns priorities to data packets and generates corresponding layer identifiers;

[0041] The synchronous sensing arbitration module, integrated in the router, receives the priority and layer identifier carried in the data packets and performs synchronous sensing arbitration for inter-layer communication.

[0042] like Figure 1 As shown, the on-chip network is a two-dimensional mesh structure with a size of 4×4. First, the original network is flattened into an ANN-like neural network. Most operations in CNNs (such as convolution and pooling) can be performed using MAC operations, resulting in a flattened CNN model. Figure 1 In (b), neurons are divided into different execution groups layer by layer. The computational and storage resources of each PE determine the maximum number of neurons in a group. Finally, in the NoC-based NN accelerator, these groups are mapped to different PEs. Common traffic patterns between adjacent PEs include one-to-one, one-to-many, many-to-one, and many-to-many.

[0043] In a further embodiment, for many-to-one and many-to-many traffic modes, a PE needs to receive input from multiple PEs in the previous layer. A PE can only begin computation after receiving all the input neurons required for the current round of computation. Due to factors such as the distance between the source and destination nodes, the number of data packets to be sent, and the congestion level of the transmission path, the time required to receive data packets from different source nodes varies, and the start time of each PE's computation depends on the last data packet received. In the case of many-to-many traffic mode, the completion time of a layer's computation depends on the last PE in that layer to complete the computation. In most cases, the last PE to receive all the required data packets will determine the completion time of that layer's computation. Therefore, the last data packet received in that layer will determine the completion time of the layer's computation. If the time for the last received data packet can be shortened, the total execution time can be reduced.

[0044] like Figure 2 As shown, PE0-PE3 are responsible for the computation of the first layer, PE4-PE5 for the computation of the second layer, and PE6 for the computation of the third layer. PE4 needs to receive input from PE0 and PE1, PE5 needs to receive input from PE2 and PE3, and PE6 needs to receive input from PE4 and PE5. It can be seen that under the synchronous awareness optimization, the last data packet of each layer is received earlier, reducing the total execution time.

[0045] This application takes into account the traffic patterns in practical applications, namely the traffic patterns in neural network accelerators based on on-chip networks, and optimizes the communication synchronization problem of many-to-one and many-to-many traffic in this application scenario.

[0046] The specific steps for assigning priorities to data packets are as follows:

[0047] The system receives valid signals from data packets, calculates the scaling factor for each processing unit, and generates a priority. The priority formula is as follows:

[0048]

[0049] in For the floor function, N total N trans N left These represent the total number of packets generated, the number of packets sent, and the number of packets remaining in one inference process of the processing unit, respectively. C is the scaling factor of the processing unit, and the formula for calculating C is:

[0050]

[0051] Where B is the bit width used for priority, which is set to 8 in this invention.

[0052] To solve the communication synchronization problem and reduce the difference in transmission delay between different nodes, we usually need to consider the length of the transmission path, the degree of congestion, etc. between different nodes. However, this invention uses a backpressure mechanism to convert indicators such as the length of the transmission path and the degree of congestion into the number of remaining data packets of the source node. The latter is a variable that is easier to quantify and control, simplifying the priority generation logic.

[0053] The priority generation module assigns a priority to each data packet that is proportional to the number of remaining data packets in the processing unit. This allows for negative feedback control over the number of remaining data packets in the source node: the number of remaining data packets in the processing unit is positively correlated with the data packet transmission delay, and the data packet transmission delay is:

[0054] T = Tqueue + ∑1 hops (Tpipeline+Tblock+Tcontention+Tlink)+Tsize

[0055] Here, Tqueue represents the packet queuing delay, Tsize is the delay determined by the packet size, Tpipeline is the delay caused by the number of router pipeline stages, Tlink is the link transmission delay, Tblock is the delay caused by downstream router congestion, and Tcontention is the delay caused by packet contention. Among these delays, Tcontention is negatively correlated with packet priority. As shown by the expression for priority, priority is proportional to the number of remaining packets. Compared with other processing units at the same layer, when there are more remaining packets, priority will increase, Tcontention will decrease, and the number of remaining packets will decrease. Conversely, when there are fewer remaining packets, priority will decrease, Tcontention will increase, and the number of remaining packets will increase, thus achieving negative feedback regulation of the number of remaining packets.

[0056] In a further embodiment, such as Figure 3 As shown, compared to nodes 6 and 7, which also belong to the second layer, node 5 has a longer average distance to the third-layer PEs (nodes 10 and 11), and a higher proportion of congested paths on its transmission path. Data packets sent by node 5 will take longer to reach their destination, causing packets to stagnate and accumulate at the source node, ultimately halting the node's computation due to limited buffer capacity. Due to the aforementioned backpressure process, node 5 will have a larger number of remaining data packets than other source nodes in this layer. Under the negative feedback mechanism of this invention, based on the number of remaining data packets, node 5 will have the highest priority, followed by node 6, and node 7 will have the lowest priority.

[0057] In the negative feedback regulation process, when there are more remaining data packets in the source node, the data packets sent by the source node will have a higher priority, and their transmission will be accelerated. This will result in a faster data packet sending rate, which in turn will lead to fewer remaining data packets in the source node, a decrease in data packet priority, and a decrease in the data packet sending rate of the source node. Through the negative feedback regulation mechanism, each source node in the same layer will have approximately the same number of remaining data packets.

[0058] For a source node at the same layer, having more remaining data packets, higher priority, and a faster packet sending rate will, in turn, reduce the number of remaining data packets and decrease the priority. This is a negative feedback process that can eventually achieve a dynamic balance, making the number of remaining data packets for each source node similar.

[0059] The backpressure mechanism in this application converts indicators such as data packet transmission path length and congestion level into the number of remaining data packets at the source node, which facilitates subsequent control. By using a negative feedback mechanism, the priority is set to be negatively correlated with the number of remaining data packets, and negative feedback adjustment is performed to control the packet sending rate of each source node, so that each source node has a similar number of remaining data packets, thereby achieving the purpose of communication synchronization.

[0060] In a further embodiment, the priority generation unit of each processing unit consists of a layer identifier register, a scaling counter, and a priority counter. The layer identifier register stores the neural network layer number that the processing unit is responsible for, and is assigned a value during the configuration phase of each processing unit; the layer identifier of the data packet is given by this register. The scaling counter is used to scale the number of data packets sent; it generates an output to the priority counter once every C data packets received, corresponding to the value in the Priority expression. The priority counter decrements by 1 each time it receives the output of the scaling counter; this count value represents the priority. The initial values ​​of the layer identifier register, scaling counter, and priority counter are assigned during the configuration phase of each processing unit. This invention uses an 8-bit priority bandwidth, i.e., a priority range of 0-255. Figure 4 This demonstrates the specific process of priority generation:

[0061] Sa, configure the initial value C of the scaling counter and the initial value of the priority counter.

[0062] When sending a data packet, the scaling counter is decremented by 1. When the scaling counter reaches 0, the initial value C of the scaling counter is reset. At the same time, the priority counter is decremented by 1, and the current count value of the counter is output, which is the priority.

[0063] Sc. When the priority counter reaches 0, reset the initial value of the priority counter.

[0064] In this invention, the 8-bit priority bandwidth is a trade-off between hardware overhead and control effect. The 8-bit priority bandwidth provides 256 priority levels. A larger priority bandwidth can provide more priority levels, obtain finer control granularity, further reduce the difference in the number of remaining data packets of each source node, help communication synchronization, and reduce inter-layer latency. However, a larger priority bandwidth will bring greater hardware overhead, which needs to be weighed. In scenarios with higher performance requirements, a larger priority bandwidth can be used to improve performance.

[0065] The synchronous sensing arbitration module includes at least five sub-arbiters that receive data packets from different directions, perform inter-layer communication synchronous sensing arbitration, select the corresponding output port according to the priority and layer identifier in the data packet, and output the arbitrated data packet.

[0066] In a further embodiment, each router contains a synchronization-aware priority arbitration module for performing synchronization-aware arbitration of data packets. The synchronization-aware priority arbitration module internally includes five sub-arbiters, each handling input requests from the five directions of North (N), South (S), East (E), West (W), and Local (L), respectively. These sub-arbiters receive the priority and layer identifier of data packets from different requesting ports, where the layer identifier indicates which layer of the neural network the data packet originated from, and output the port from which arbitration was successful.

[0067] like Figure 5 As shown, Figure 5 (a) Initial requests from different input ports are given, where requests with the same line indicate they originate from the same layer. Input ports N, S, and W request output port E, and input ports E and L request output port S. In the first step, for output port E, input ports N and S need to compete because their packets originate from the same layer (assuming input port N has higher priority), while input port W can directly proceed to the second step because its packets originate from a different layer. For output port S, input ports E and L need to compete because their packets originate from the same layer (assuming input port L has higher priority). The authorized input ports generated after step 1 are as follows: Figure 5 As shown in (b). In the second step, for output port E, a polling strategy is used to arbitrate between input ports N and W, and finally input port W is selected. The final authorized port is as follows. Figure 5 As shown in (c).

[0068] The specific steps for inter-layer communication synchronous sensing arbitration are as follows:

[0069] Analyze the priority and layer identifier in the data packets; for data packets originating from the same layer, select the data packet with the highest priority; for data packets originating from different layers and data packets with the same priority within the same layer, use a round-robin arbitration algorithm to give the data packets the same priority.

[0070] In a further embodiment, the arbitration process of each sub-arbitrator consists of two steps. First, for packets originating from the same layer, the module selects the packet with the highest priority. Second, for packets originating from different layers and packets with the same priority as those in the first step, the module uses a round-robin arbitration algorithm to give these packets the same priority.

[0071] This arbitration process avoids starvation. First, the algorithm arbitrates among packets from the same layer, using priority information as the arbitration criterion. By comparing priorities, the packet with the most remaining packets at its source node within its layer is selected. This means that when a packet lingers in the router, its transmission priority is relatively higher compared to other packets from the same layer, preventing it from remaining on the current router for an extended period. Next, the algorithm selects among packets from different layers with the same priority. In this step, the arbitration strategy is round-robin, meaning all packets have equal priority, thus eliminating starvation. Based on this analysis, starvation is not an issue throughout the entire arbitration process.

[0072] In a further embodiment, to verify the effectiveness of the arbitration module proposed in this invention in improving the performance of neural network applications, the communication architecture was simulated on the CNN-Noxim simulation platform. The simulation adopted an 8*8 two-dimensional network topology with a buffer depth of 8 flits, 3 virtual channels per input port, and a data packet size of 8 flits transmitted in the network. The routing module used in the simulation was XY-dimensional routing, and the tested neural networks included LeNet, Steering ControlCNN, and VGG16. Considering the limited computing and storage resources on each PE, VGG16 only performed the mapping of the first three layers. These three networks represent small, medium, and large-scale networks, respectively. This simulation mainly verifies whether the perceptual optimization made by the arbitration module of this invention for the communication synchronization problem can reduce the overall inference time of the neural network.

[0073] Figure 6The figure shows the inference time for different arbitration strategies under different neural networks. Five mapping methods were used for each network: dirx (arranged along the x-direction), diry (arranged along the y-direction), and three random mappings. The average is the average value of these mappings. In the figure, CSAP represents the arbitration strategy proposed in this invention, namely the Communication Synchronization-aware Arbitration Policy. Round-Robin, FIFO, and GA represent three arbitration strategies based on polling arbitration, local age, and global age, respectively. The arbitration strategy based on global age is considered one of the best performing arbitration strategies. It is easy to see that compared with other arbitration strategies, this invention achieves significant improvements in small and medium-sized networks. Its performance is slightly worse than the global age-based arbitration strategy in large networks because the large network scale leads to long execution times, making global age an important arbitration indicator. However, due to the complexity of maintaining a global timestamp throughout the system, implementing a global age strategy in hardware is impractical.

[0074] Furthermore, router arbitrators employing round-robin arbitration, local age-based arbitration, and the present invention were implemented using Verilog, and their hardware overhead was evaluated on a 28nm technology node using Synopsys Design Compiler. The hardware implementation overhead for different arbitration strategies is shown below:

[0075] When using a polling strategy, the frequency is 1GHz, the area is 120.33μm2, accounting for 0.83% of the area in the router, and the power consumption is 0.139mW, accounting for 5.67% of the power consumption in the router.

[0076] When using the local age policy, the frequency is 1GHz, the area is 405.09μm2, accounting for 2.75% of the area in the router, and the power consumption is 0.191mW, accounting for 7.63% of the power consumption in the router.

[0077] When using the strategy of this invention, the frequency is 1GHz, the area is 708.75μm2, accounting for 4.71% of the area in the router, and the power consumption is 0.29mW, accounting for 11.1% of the power consumption in the router.

[0078] Compared to the polling strategy, this invention increases the area overhead in the router by only 3.88%, and compared to the local age-based strategy, it increases it by only 1.92%. In this invention, the priority generation unit within each processing unit consists of only two levels of counters, and the hardware overhead is negligible compared to the entire processing unit.

[0079] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A synchronization-aware arbitration module for a network-on-chip based neural network accelerator, the module comprising: include: The priority generation module, integrated into the processing unit, assigns priorities to data packets and generates corresponding layer identifiers; The synchronous sensing arbitration module, integrated in the router, receives the priority and layer identifier carried in the data packets and performs synchronous sensing arbitration for inter-layer communication. The priority generation module includes: The layer identifier register stores the neural network layer number that the processing unit is responsible for. It is assigned a value during the configuration phase of each processing unit. The layer identifier of the data packet is given by this register. The scaling counter sends an effective signal to the priority counter once for every C received data packets, and is used to scale the number of data packets that have been sent, corresponding to wherein C is the scaling coefficient of each processing unit, N trans is the number of data packets that have been sent by each processing unit, N total is the total number of generated packets for each inference process of each processing unit, and B is the bit width of the priority usage. The priority counter decrements by 1 each time it receives the output of the scaling counter; this count value represents the priority. The priority generation module's workflow is as follows: Sa, the value of the configuration layer identification register is L, the initial value of the scaling counter is C and the initial value of the priority counter is N, wherein N corresponds to When sending a data packet, the scaling counter is decremented by 1. When the scaling counter reaches 0, the initial value of the scaling counter is reset to C. At the same time, the priority counter is decremented by 1. The current count value of the counter is output, which is the priority of the data packet. The value of the layer identifier register is read, which is the layer identifier of the data packet. Sc. When the priority counter reaches 0, reset the initial value of the priority counter to N; The specific steps for inter-layer communication synchronous sensing arbitration are as follows: Priority and layer identifier in received data packets; For data packets originating from the same layer, select the data packet with the highest priority. For data packets originating from different layers and data packets with the same priority within the same layer, a round-robin arbitration algorithm is used to give these data packets the same priority.

2. The synchronous sensing arbitration module for on-chip network-based neural network accelerators according to claim 1, characterized in that, The synchronous sensing arbitration module includes at least five sub-arbiters, each corresponding to an output direction. It performs inter-layer communication synchronous sensing arbitration on all data packets requesting that output direction and outputs the successfully arbitrated data packet according to the priority and layer identifier in the data packet.

Images