A high real-time approximate on-chip network oriented to convolutional neural network acceleration

Abstract

The application relates to the technical field of approximate network-on-chip, and discloses a high real-time approximate network-on-chip for convolutional neural network acceleration, which combines the data distribution of a neural network, proposes an approximate compression and segmented recovery scheme based on data grading, realizes reduction of communication overhead while maintaining neural network inference accuracy, and proposes a quality control model which can quickly search for neural network layer thresholds satisfying a quality loss constraint, ensuring that the network maintains accuracy while performing approximate communication; further, a congestion-aware threshold dynamic adjustment scheme is proposed, which dynamically adjusts the approximate threshold of each layer by sensing the congestion in the network, so as to realize real-time optimization of the data compression rate and effectively alleviate the congestion problem in the communication network; in combination with the activation data distribution characteristics in the network after approximate communication, the approximate calculation module in the calculation array is optimized, the overall efficiency of the calculation array is improved, and calculation acceleration is realized.

Description

Technical Field

[0001] This invention relates to the field of approximate on-chip network technology, and more particularly to a high real-time approximate on-chip network for accelerating convolutional neural networks. Background Technology

[0002] Since the advent of Moore's Law, each new generation of manufacturing processes has reduced the area of ​​a single transistor, allowing for a greater number of transistors within the same area. Even with the continuous increase in computing power per core, limitations such as the power and memory walls prevent a single core from reaching infinite size, thus giving rise to multi-core systems-on-a-chip (SoCs). SoCs solve the problem of insufficient computing power in high-performance computing applications. These high-performance multi-core processors need to provide sufficiently large transmission bandwidth while maintaining extremely low transmission latency. Therefore, on-chip networking (BON) has become an efficient on-chip communication architecture connecting processor cores. SoCs interconnect nodes through routers, with multiple communication data streams time-division multiplexed on these links, achieving high scalability and large transmission bandwidth.

[0003] Research shows that neural networks possess strong fault tolerance, which makes the use of approximation techniques during inference possible. However, as the scale of neural networks increases, the number of parameters grows exponentially, placing a heavy burden on the communication and computation of on-chip network-based multi-core neural network accelerators. This leads to a sharp increase in total execution time, making it unsuitable for real-time applications. Currently, research on on-chip network-based neural network accelerators, both domestically and internationally, mainly focuses on approximate communication. These works, based on application quality constraints, reduce the communication load on the on-chip network by using lossy data compression to lower the average latency and meet real-time requirements. However, these works do not consider the nature of the application itself, such as the data distribution, thus failing to effectively increase the proportion of compressed data packets; furthermore, they do not coordinate computational optimization with approximate communication, limiting further reductions in total execution time.

[0004] Approximate communication is defined as the application of approximation techniques in parallel systems, aiming to reduce communication between peers (PEs). Existing NoC-based approximate communication schemes can be divided into two categories: approximating the entire data in flits and approximating the lowest bits of the data in flits. Approximating the entire data in flits often involves discarding flits at fixed intervals at the source node and recovering them at the destination node using linear interpolation. This approximation and recovery scheme limits the overall data compression rate to no more than 50%. Furthermore, the fixed-interval discarding may lead to the loss of activation values ​​that significantly impact neural network inference results, thus reducing model accuracy. Approximating the lowest bits of the data in flits often employs pattern compression and recovery. However, modern neural network accelerators frequently use low-bit quantization, and truncating the lowest bits introduces significant errors. Under quality constraints, the truncation width is limited, resulting in low data compression rates and failing to alleviate communication congestion.

[0005] During convolutional neural network (CNN) inference, a large number of concurrent multiply-accumulate operations occur, and the power consumption and latency of the multiply-accumulate units have become one of the bottlenecks limiting the performance improvement of neural network accelerators. The fault-tolerant nature of CNNs makes it possible to use approximate calculations during inference. Provided the quality loss of the neural network is acceptable, the power consumption and latency of the neural network accelerator can be reduced by introducing approximate units such as approximate multipliers and approximate adders. In the computational power consumption of CNNs, the multiplier accounts for 90% of the computational power consumption. Therefore, introducing approximations for the multiplier can effectively reduce the computational power consumption of the neural network accelerator and improve its performance. Existing approximate multiplier designs mainly fall into two categories: one type reduces the power consumption of the multiplier by reducing the voltage without changing the multiplier logic; the other type introduces approximations by simplifying certain circuit units of the accurate multiplier to optimize power consumption and latency. Reducing multiplier power consumption by voltage scaling increases critical path delay, failing to meet real-time requirements. Conversely, simplifying the circuit logic of an exact multiplier by introducing approximations without considering the data distribution characteristics of neural networks can easily lead to under- or over-approximation. Under-approximation fails to fully utilize the fault-tolerant nature of neural networks, while over-approximation results in excessive quality loss and unacceptable inference accuracy. Furthermore, current designs for approximate multipliers used for neural network acceleration lack organic integration with communication.

[0006] For example, Chinese Patent Publication No. CN117411561A discloses an optical network-on-a-chip (ONA) communication method based on approximate calculation and an ONA itself. This ONA communication method applies approximate calculation technology to ONA and combines it with power control to achieve reliable low-power communication. Compared with traditional communication, this method makes reasonable use of the inherent robustness of the application, reduces data movement during communication through approximate calculation technology, and sets the required power for communication. While meeting reliability requirements, it avoids power waste and improves the power efficiency and execution speed of ONA.

[0007] For example, Chinese patent CN109547263B discloses an on-chip network optimization method based on approximate calculation, which includes a data trimmer, a data restorer, a traffic predictor, a global controller, and a local controller. The data trimmer trims data packets before they are injected into the network interface, shortening the packet length. The data restorer recovers lost data after receiving the trimmed packets. The traffic predictor predicts the data traffic in the next control interval based on past node communication data. The global controller calculates the approximate optimal configuration for each node from a global perspective, based on global information and user quality requirements, and sends control information to each node. The local controller configures the data loss rate for each packet waiting to be injected into the network based on the received control information. This method can optimize the performance and power consumption of the on-chip network at a relatively low cost, without violating user requirements for output quality.

[0008] The above patents suffer from the problems described in this background: neither of them addresses how to effectively compress data using inter-layer data from convolutional neural networks, nor do they organically integrate the approximate computation module for neural network acceleration with communication. To solve this problem, this invention proposes a high-real-time approximate on-chip network for accelerating convolutional neural networks. Summary of the Invention

[0009] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0010] In view of the problems existing in the current high real-time approximate on-chip network for accelerating convolutional neural networks, the present invention is proposed.

[0011] Therefore, the purpose of this invention is to provide a high real-time approximate on-chip network for accelerating convolutional neural networks.

[0012] To address the aforementioned technical problems, this invention provides a high real-time approximate on-chip network for accelerating convolutional neural networks: a global threshold dynamic controller at the central node and a local controller in each node;

[0013] The global threshold dynamic controller is used to monitor congestion in the network and adjust the approximate threshold of each node according to the congestion situation.

[0014] The local controller includes an approximation module and a recovery module. The approximation module approximates data packets according to a global threshold, and the recovery module recovers data packets based on the approximation information in the header flit of the data packets.

[0015] Each node's approximate multiply-accumulate unit employs an approximate calculation module that matches the distribution characteristics in the recovered data packet. The approximate calculation module includes a sparse rearrangement module, a shift multiply-accumulate module, and an approximate multiply-accumulate module.

[0016] As a preferred embodiment of the high real-time approximate on-chip network for accelerating convolutional neural networks described in this invention, the sparse rearrangement module is used to perform zero-value sparsification on the input data to obtain activation data, and to rearrange the non-zero data.

[0017] The shift-multiply-accumulate module identifies the recovered non-zero data and performs multiplication by shifting.

[0018] The approximate multiply-accumulate module calculates the active data that has not been approximated during the approximate communication process;

[0019] The results from the shift-multiply-accumulate module and the approximate multiply-accumulate module are reduced, and the output data is used for transmission at the next level.

[0020] As a preferred embodiment of the high real-time approximate on-chip network for accelerating convolutional neural networks described in this invention, the global threshold controller is used to implement a congestion-aware dynamic threshold adjustment scheme.

[0021] The congestion-aware threshold dynamic adjustment scheme dynamically adjusts the approximate threshold of each node by sensing the congestion situation in the network.

[0022] The congestion-aware threshold dynamic adjustment scheme includes:

[0023] The global threshold controller initializes the node number of each layer and records the network layer number corresponding to different nodes and the number of nodes in each layer. The global threshold controller obtains the congestion status of each layer node. The congestion status of each layer node is measured by the free slot occupancy rate of the local buffer. If a node in a certain layer is congested, the congest flag of that node is set to 1, and the number of nodes in each layer that are congested is counted. The global threshold controller calculates the proportion of the number of nodes in each layer that are congested to the total number of nodes in that layer to evaluate which layer is the most congested. If the most congested layer has the same configured approximate threshold, the second most congested layer is found, and the selected configuration information is determined by the different configured approximate thresholds of the second most congested layer.

[0024] As a preferred embodiment of the high real-time approximate on-chip network for accelerating convolutional neural networks described in this invention, the shift-multiply-accumulate module consists of three position registers, three weight registers, an adder, and a shifter. The shift-multiply-accumulate module receives the output of the sparse rearrangement module, which is less than the approximate threshold, as input.

[0025] As a preferred embodiment of the high real-time approximate on-chip network for accelerating convolutional neural networks described in this invention, the approximate communication includes hierarchical data compression and recovery, and the logic of the hierarchical data compression includes:

[0026] Set an approximation threshold T0~Tn. If the activation data of a layer is less than T0, then set the approximation level of the activation data of that layer to 0.

[0027] If the activation data of this layer is between T0 and T1, then the approximate level of the activation data of this layer is set to 1, and so on. For activation data that is greater than Tn-1 but less than Tn, the approximate level of the activation data of this layer is set to n-1.

[0028] The position and approximation level of the activation data to be approximated are recorded in the header flit to generate approximation information, and the activation data to be approximated is deleted from the data packet. No approximation is performed on activation data greater than Tn.

[0029] As a preferred embodiment of the high real-time approximate on-chip network for accelerating convolutional neural networks described in this invention, the recovery logic includes:

[0030] After the data packet is transmitted to the destination node, the recovery module extracts the approximate information in the header flit and restores the activation data to the threshold of the left boundary within the segment based on the approximate level of the activation data.

[0031] As a preferred embodiment of the high real-time approximate on-chip network for accelerating convolutional neural networks described in this invention, the approximation threshold is obtained through a quality model;

[0032] The quality model is used to determine an approximate threshold that satisfies the quality loss constraint for each layer of the network. The quality model is executed on a GPU and generates configuration information for the approximate threshold.

[0033] The execution process of the quality model includes:

[0034] Sort the activation data for each layer, extract the data values ​​located in the top 20%, top 40%, top 60%, and top 80% of the activation data distribution for that layer, and find the data value that can be represented in the form of 2^m + 2^h + 2^k as the approximate threshold for that layer. After completing the selection of the approximate threshold for each layer, iterate through all the combinations of approximate thresholds for each layer, and select the N configurations that maximize the network data compression rate from the configurations that satisfy the quality loss constraints.

[0035] As a preferred embodiment of the high real-time approximate on-chip network for accelerating convolutional neural networks described in this invention, the distribution characteristics of the activation data in the network after approximate communication are obtained by analyzing the structure and data characteristics of the convolutional neural network.

[0036] As a preferred embodiment of the high real-time approximate on-chip network for accelerating convolutional neural networks described in this invention, the approximation module includes a data FIFO, an approximate data FIFO, an approximate threshold register, a comparator, and an approximation level encoder.

[0037] The recovery module includes a flit decoder, an approximation level register, an approximation threshold register, and a data FIFO;

[0038] The flit decoder extracts the approximate information from the flit header and stores it in the corresponding position in the approximate level register. The data selector selects different data based on the approximate level information and stores them in the data FIFO.

[0039] The beneficial effects of this invention are as follows: This invention proposes a hierarchical approximate compression and segmented recovery scheme based on the data distribution in neural networks. By performing hierarchical data compression, it achieves maximum preservation of neural network inference accuracy while reducing communication overhead. A quality control model is also proposed, which can quickly search for the thresholds of each layer of the neural network that satisfy the quality loss constraint, ensuring that the network maintains accuracy during approximate communication. Based on the quality control model, a congestion-aware dynamic threshold adjustment scheme is further proposed. This scheme dynamically adjusts the approximate threshold of each layer by sensing the congestion situation in the network, achieving real-time optimization of the data compression rate and effectively alleviating congestion problems in the communication network. Combining the distribution characteristics of activated data in the network after approximate communication, the approximate calculation module in the computing array is optimized, improving the overall efficiency of the computing array and achieving computational acceleration. This method significantly shortens the total execution time of neural network inference and improves the real-time performance and overall system performance. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0041] Figure 1 This is a diagram of the overall architecture of a fusion approximate on-chip network for high real-time performance of convolutional neural network acceleration according to the present invention.

[0042] Figure 2 This is a diagram of the activation data distribution of AlexNet, a high real-time approximation of an on-chip network for accelerating convolutional neural networks, according to the present invention.

[0043] Figure 3 This is a distribution diagram of activation data in a certain layer after approximate communication of AlexNet, a high real-time approximate on-chip network for accelerating convolutional neural networks, according to the present invention.

[0044] Figure 4 This is a sparse rearrangement module diagram of a high real-time approximate on-chip network for accelerating convolutional neural networks according to the present invention.

[0045] Figure 5 This is a diagram of a shift-multiply-accumulate module for a high real-time approximate on-chip network for accelerating convolutional neural networks according to the present invention.

[0046] Figure 6 This is a diagram of an approximate multiply-accumulate module for a high real-time approximate on-chip network for accelerating convolutional neural networks according to the present invention.

[0047] Figure 7 This is an approximate module diagram of a high real-time approximate on-chip network for accelerating convolutional neural networks according to the present invention;

[0048] Figure 8 This is a diagram of a recovery module for a high real-time approximate on-chip network for accelerating convolutional neural networks, according to the present invention.

[0049] Figure 9 This is a comparison chart of the total execution time of a high real-time approximate on-chip network for accelerating convolutional neural networks according to the present invention;

[0050] Figure 10 This is a comparison chart of data compression rates for a high real-time approximate on-chip network for accelerating convolutional neural networks, as presented in this invention.

[0051] Figure 11 This is a comparison of the computation time of a high real-time approximate on-chip network for accelerating convolutional neural networks according to the present invention.

[0052] Figure 12 This is a comparison diagram of communication time for a high real-time approximate on-chip network for accelerating convolutional neural networks according to the present invention. Detailed Implementation

[0053] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0054] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention can also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0055] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0056] Example

[0057] like Figure 1 As shown, a high real-time approximate on-chip network for accelerating convolutional neural networks includes a global threshold dynamic controller at the central node and a local controller in each node.

[0058] The global threshold dynamic controller is used to monitor congestion in the network and adjust the approximate threshold of each node according to the congestion situation.

[0059] The local controller includes an approximation module and a recovery module. The approximation module approximates data packets according to a global threshold, and the recovery module recovers data packets based on the approximation information in the header flit of the data packets.

[0060] Each node’s approximate multiply-accumulate unit uses an approximate calculation module that matches the distribution characteristics in the recovered data packet. The approximate calculation module includes a sparse rearrangement module, a shift multiply-accumulate module, and an approximate multiply-accumulate module.

[0061] The sparse rearrangement module prefetches multiple valid activations, then detects the positions where the activation is zero and cleans the data; the shift-multiply-accumulate module identifies the recovered non-zero values ​​and performs multiplication by shifting; the approximate multiply-accumulate module uses an improved Drum multiplier to calculate the activation data that was not approximated during the approximate communication process; finally, the results of the two multiply-accumulate modules are merged into the final result for transmission to the next level.

[0062] Existing research shows that, in different convolutional neural network structures and different datasets, the weight data and activation data of each layer of the convolutional neural network mostly exhibit Gaussian-like distribution or truncated Gaussian distribution characteristics. This distribution characteristic is mainly due to the addition of a normalization layer in each layer of the neural network, which adjusts the distribution of activation data and weight data of each layer to a state of symmetry about the zero axis.

[0063] Meanwhile, the ReLU activation function is usually applied after the convolutional layer. This activation function truncates all negative values ​​to zero, which makes the activation data exhibit the characteristics of a truncated Gaussian distribution and leads to a large number of zero values, making the output data highly sparse. Inspired by the pruning principle of neural networks, research shows that when weights or activation values ​​close to zero are directly set to zero, the impact on the inference accuracy of the network is very limited.

[0064] Therefore, we propose a hierarchical compression and segmented recovery scheme based on the amplitude of active data. By performing compression processing on active data at different levels according to amplitude, we achieve efficient data compression. And by using a segmented recovery method, we minimize communication overhead while ensuring network accuracy.

[0065] Before introducing the principles of this data compression and recovery scheme, it is necessary to first explain the composition of data packets during communication between processing units (PEs) in a Network on-Chip (NoC). A data packet typically consists of three parts: a header flit, a body flit, and a tail flit. The header flit contains key information about the data packet, including the flit type marker, source node coordinates, destination node coordinates, virtual channel sequence number, routing algorithm, and neuron index, which guides the transmission and processing of the data packet. The body flit carries the main data payload, including the flit type, sequence number, and specific data information. The tail flit can be empty or consistent with the data information in the body flit, used to indicate the end of the data packet.

[0066] Typically, a complete data packet consists of a header flit and a tail flit, with N individual flits in between carrying the actual data information. The header flit has been extended by adding information about the position of approximate data within the data packet and its approximation level. This improvement allows the destination node to effectively recover the received approximate data based on this additional information.

[0067] Taking the activation output of a certain layer in a neural network as an example, a specific scheme based on hierarchical data compression and segmented recovery is proposed. The principle of this scheme is as follows: First, a series of approximate thresholds T0~Tn are set. For the activation data of this layer, if the activation data is less than T0, its approximation level is set to 0; if the activation data is between T0 and T1, its approximation level is set to 1, and so on. For activation data that is greater than Tn-1 but less than Tn, its approximation level is set to n-1.

[0068] The position of the approximated volume flit and its corresponding approximation level are recorded in the header flit, while the approximated active data is deleted from the data packet. Subsequent flits are moved forward, thereby shortening the length of the data packet and improving transmission efficiency. For active data whose amplitude exceeds the approximation threshold range, i.e., active data greater than Tn, no approximation processing will be performed, and its original accuracy will be preserved.

[0069] After the data packet is transmitted to the destination node, the recovery module extracts the approximate information from the header `flit` and performs segmented data recovery based on this information. Specifically, the recovery process involves restoring the active data to the threshold of the left boundary within each segment based on its approximation level. For example, active data with an approximation level of 0 will be restored to 0, active data with an approximation level of 1 will be restored to T0, and so on, thus completing the reconstruction of the approximate data. This process ensures a balance between the compression efficiency of the active data and the recovery accuracy, maximizing the compression ratio while minimizing the impact on network inference accuracy.

[0070] To achieve the aforementioned approximate communication based on data hierarchical compression and recovery schemes, a quality model is proposed to quickly determine approximate thresholds that satisfy quality loss constraints for each layer of the network. This process is executed on the GPU and generates threshold information called a "configuration table".

[0071] The specific process is as follows: First, execute a complete inference process. Then, sort the activation data of each layer and extract the data values ​​located in the top 20%, top 40%, top 60%, and top 80% of the activation data distribution of that layer. The reason for choosing these proportions is that experimental results show that under the data hierarchical compression and recovery scheme, most neural networks can only process approximately 80% of the data at most in a certain layer. Dividing into four levels facilitates congestion awareness and dynamic adjustment during network operation.

[0072] Next, the data value that is closest to these data values ​​and can be represented in the form 2^m + 2^h + 2^k is found as the approximation threshold for that layer. Choosing these approximation thresholds helps to accelerate computation and improve energy efficiency. The quality loss of a neural network is closely related to the approximation rate of the data. When a layer becomes congested, choosing a larger approximation threshold means that the data compression rate (i.e., the approximation rate) of that layer will be higher. However, in order to meet the overall quality loss constraints, it is necessary to reduce the data approximation rate of other layers, that is, to reduce the approximation thresholds of these layers.

[0073] Then, after selecting the approximate threshold for each layer, considering that not all threshold combinations can satisfy the quality loss constraint, it is necessary to traverse all possible approximate threshold combinations to find the optimal solution. To reduce the computational overhead of this traversal, the search space for approximate thresholds is reduced by merging network layers that occupy fewer nodes. For example, when a network has 19 layers, theoretically 4^18 possible combinations need to be traversed (the last layer does not undergo approximation, so only 18 layers need to select approximate thresholds), which leads to a huge time overhead. However, merging layers that occupy fewer nodes significantly reduces the search space. In addition, this layer binding operation can make the number of nodes adjusted more evenly during dynamic adjustment, thereby making the network load distribution more balanced during inference on NoC.

[0074] In practical applications, the neural network quality loss threshold is set to 2%. After traversing to obtain the approximate threshold configuration, the approximate communication and approximate computation are further combined to infer the neural network on the C++ platform. The approximate multiply-accumulate unit proposed in the invention is used to replace the exact approximate multiply-accumulate unit for inference. num threshold configurations that satisfy the quality loss constraint under the combination of computation and communication are saved.

[0075] When deploying neural networks on NoC for inference, the traffic at each layer of the network changes dynamically over time. To effectively alleviate communication congestion, a congestion-aware dynamic threshold adjustment scheme is proposed. This scheme embeds a global dynamic threshold controller in the global control node to sense the congestion status of each layer's nodes in real time and dynamically adjusts the approximate threshold of each layer's nodes based on these conditions, thereby mitigating potential congestion problems during system operation.

[0076] The controller first initializes the number of each node and records the network layer number corresponding to different nodes and the number of nodes in each layer. Then, the controller obtains the congestion status of each local node. The congestion status of a local node is measured by the free slot occupancy rate of the local buffer. If a node becomes congested, the system sets the congest flag of that node to 1 and counts the number of congested nodes in each layer. For example, if there are 3 congested nodes in the first layer and 2 congested nodes in the second layer, the controller then calculates the proportion of the number of congested nodes in each layer to the total number of nodes in that layer to evaluate which layer is the most congested. Then, using a greedy algorithm, if multiple configuration thresholds are the same for the most congested layer, the next most congested layer is identified, and the final selected configuration information is determined by the different configuration thresholds of that layer, and so on.

[0077] This congestion-aware dynamic threshold adjustment scheme can adjust the approximate thresholds of each layer in real time during network operation based on the relative congestion status of different layers. This significantly alleviates congestion problems in different applications and mapping scenarios, thereby improving system speed and shortening the total execution time of neural network inference. Once the optimized threshold configuration is obtained, the global control node sends it to each local node. Upon receiving the new approximate threshold, the approximation and recovery modules in the local nodes perform hierarchical approximation and segmented recovery operations based on the updated threshold information to ensure efficient network operation.

[0078] Traditional approximate neural network accelerators typically only consider one aspect of approximate communication or approximate computation, failing to organically combine the two and thus failing to further reduce the overall execution time of neural network inference. To address this limitation, an approximate computation module with low latency and low power consumption is designed, leveraging the data distribution characteristics after approximate communication. This module can be organically combined with approximate communication to achieve computational acceleration. As mentioned earlier, the inter-layer activation data of convolutional neural networks typically exhibits a Gaussian-like distribution or a truncated Gaussian distribution (e.g., ...). Figure 2 As shown), after processing with the proposed hierarchical compression and segmented recovery scheme, the active data recovered by the approximate communication at the destination node exhibits a discrete distribution below the threshold and a continuous distribution above the threshold (as shown). Figure 3 (As shown).

[0079] Furthermore, since the proportion of 0 values ​​in the restored activated data has increased significantly, based on this characteristic, the approximate calculation module proposed in this invention consists of three core sub-components: a sparse rearrangement module, a shift-multiply-accumulate module, and an approximate multiply-accumulate module.

[0080] The main function of the sparse rearrangement module is to perform zero-value sparsification on the input data and rearrange the non-zero data to improve computational efficiency; the shift-multiply-accumulate module is responsible for processing data that has been approximated in approximation communication, while the approximation-multiply-accumulate module uses an improved Drum multiplier to calculate data that has not been approximated in approximation communication, to further reduce computational power consumption; the sparse rearrangement module (such as...) Figure 4 As shown, a batch of activation data is pre-extracted in one cycle, zero values ​​are filtered out by a comparator, and then the data that has been approximately recovered and the data that have not been approximated are output to the corresponding approximation calculation modules through two paths to complete the calculation.

[0081] Shift-multiply-accumulate module (e.g.) Figure 5 (As shown) This module consists of three position registers, three weight registers, an adder, and a shifter. During operation, it receives an output from the sparse rearrangement module that is less than the approximate threshold as input. First, it judges specific bits of the input. Since the set approximate threshold conforms to the form 2^m + 2^h + 2^k, it only needs to check if bits m, h, and k are 1. If they are 1, the corresponding weight register will accumulate the weight corresponding to that data. When the shift-multiply-accumulate module receives the completion signal from the sparse rearrangement module, the values ​​in the three weight registers are shifted and accumulated, finally outputting the result.

[0082] For activated data that is not approximated, these numbers are often large. In binary multiplication, the higher-order bits of the input data have a decisive impact on the accuracy of the final calculation result; their importance far exceeds that of the lower-order bits. Therefore, an approximate multiplication-accumulation module (such as...) was designed, drawing inspiration from the design concept of the DRUM multiplier. Figure 6 As shown, this module performs the calculation of this part of the data. It only sends the high K bits of the activation data, including the first 1, to the approximation calculation module. Through experiments, it was found that for most networks, the minimum value of the threshold of each layer is often greater than 256. Therefore, when the approximation multiply-accumulate module detects the first 1 of the activation value, it only needs to detect the high 8 bits. When K is fixed at 6, the inference accuracy loss of the approximation on-chip network is within 2% when used to accelerate most convolutional neural networks. At the same time, the approximation calculation module can not only skip the zero value operation, but also perform two parallel calculations of the approximation and non-approximation values ​​in the approximation communication process, and achieves significant computational acceleration with small additional overhead.

[0083] Figure 7This is the approximation module in the local node, which includes a data FIFO, an approximate data FIFO, an approximation threshold register, a comparator, and an approximation level encoder. First, we buffer the data in the storage module (Buffer) using the data FIFO. As long as the FIFO is not empty, we read from it and compare the read data with the four thresholds in the approximation threshold register. Then, we encode (combine) the comparison result into an approximation level, which is stored in the approximation level register. Simultaneously, we use a bitwise OR operation on the approximation level to control the data selector, which can discard or save approximate data to the approximation data FIFO. The data in the unapproximated volume flit is composed of the valid data from the approximation data FIFO.

[0084] Figure 8 This is the recovery module in the local node. This module includes a flit decoder, an approximation level register, an approximation threshold register, and a data FIFO. The flit decoder extracts the approximation information from the header flit and stores it in the corresponding location in the approximation level register; then, the data selector selects different data based on the approximation level information and stores it in the data FIFO.

[0085] The beneficial effects of this invention in actual simulation:

[0086] To verify the effectiveness of the proposed architecture in reducing the total inference execution time of neural network accelerators, simulations were performed on the CNN-Noxim platform. We set NoC as an 8*8 2D-mesh structure, with each data packet containing 8 fragments, each fragment being 36 bits. The neural network mapping on NoC uses random mapping, and the routing algorithm is XY routing. The benchmark used was CIFAR10-based convolutional neural networks including AlexNet, VGG11, VGG13, VGG16, and ResNet8. All networks used 16-bit quantization, with an accuracy loss of no more than 2%. Comparisons were made with state-of-the-art approximate communication frameworks including ACDC_ABDTR, ACDC_APPROX, and Neu_NoC, as well as with sparse computation neural network accelerators. The proposed framework is FAS-NoC.

[0087] The comparison dimensions include total execution time and data compression ratio (approximation ratio) (e.g., ...). Figure 9 and Figure 10(As shown). The total execution time of different architectures is normalized for NO_APPROX without approximation techniques; a higher data compression ratio indicates that more flits are discarded during approximation communication, resulting in lower communication latency. The results show that the proposed FAS-NoC has a significant effect on reducing the total execution time and improving the data compression ratio. This is because APPROX_NoC uses truncated approximation and employs a pattern matching mechanism to compress high-bit data. However, it uses 16-bit quantization for weights and activations during neural network inference. This results in a smaller truncation width while maintaining less than 2% data precision loss, and also limits the compression of high-bit data, leading to a low data approximation rate.

[0088] The ACDC_ABDTR scheme does not monitor the values ​​of transmitted data and discards fragments at fixed intervals, potentially leading to the loss of some critical large values. Under the constraint of a 2% quality loss, the number of packets that can be discarded is extremely limited. Sparse computation schemes do not consider communication optimization, resulting in limited optimization of their execution time.

[0089] In addition, to demonstrate the invention's optimization of communication and computation latency, the communication and computation times of the first few convolutional layers of VGG16 were statistically analyzed. Figure 11 and Figure 12 This result demonstrates that the proposed FAS-NoC not only alleviates communication latency but also accelerates computation. Compared with sparse computation accelerators, it further reduces computation latency by approximating some activations in the 0-T0 range as 0, which further increases the sparsity of the network and thus saves more computation time.

[0090] This invention significantly reduces the total execution time of neural network inference by organically combining approximate communication and approximate computation.

[0091] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A high real-time approximate on-chip network for accelerating convolutional neural networks, characterized in that: include: A global threshold dynamic controller at the central node and a local controller in each node; A global threshold dynamic controller is used to monitor congestion in the network and adjust the approximate threshold of each node based on the congestion situation. The global threshold controller implements a congestion-aware dynamic threshold adjustment scheme. This scheme dynamically adjusts the approximate threshold of each node by sensing congestion in the network. The congestion-aware dynamic threshold adjustment scheme includes: The global threshold controller initializes the node number of each layer and records the network layer number corresponding to different nodes and the number of nodes in each layer. The global threshold controller obtains the congestion status of each layer node. The congestion status of each layer node is measured by the free slot occupancy rate of the local buffer. If a node in a certain layer is congested, the congest flag of that node is set to 1, and the number of nodes in each layer that are congested is counted. The global threshold controller calculates the proportion of the number of nodes in each layer that are congested to the total number of nodes in that layer to evaluate which layer is the most congested. If the most congested layer has the same configured approximate threshold, the second most congested layer is found, and the selected configuration information is determined by the different configured approximate thresholds of the second most congested layer. The local controller includes an approximation module and a recovery module. The approximation module approximates data packets according to a global threshold, and the recovery module recovers data packets based on the approximation information in the header flit of the data packets. Each node's approximate multiply-accumulate unit employs an approximate calculation module that matches the distribution characteristics of the recovered data packet. This approximate calculation module includes a sparse rearrangement module, a shift-multiply-accumulate module, and an approximate multiply-accumulate module. The sparse rearrangement module performs zero-value sparsification on the input data to obtain active data and rearranges the non-zero data. The shift-multiply-accumulate module identifies the recovered non-zero data and performs multiplication through shifting. The approximate multiply-accumulate module calculates the active data that was not approximated during the approximate communication process. The results from the shift-multiply-accumulate module and the approximate multiply-accumulate module are reduced, and the output data is used for transmission at the next level. The approximate communication includes hierarchical data compression and recovery, and the logic for hierarchical data compression includes: An approximation threshold T0~Tn is set. If the activation data of a layer is less than T0, the approximation level of the activation data of that layer is set to 0; if the activation data of a layer is between T0 and T1, the approximation level of the activation data of that layer is set to 1, and so on. For activation data greater than Tn-1 but less than Tn, the approximation level of the activation data of that layer is set to n-1. The position and approximation level of the activation data to be approximated are recorded in the header flit to generate approximation information, and the approximated activation data is deleted from the data packet. No approximation processing is performed on activation data greater than Tn. The approximate threshold is obtained through a quality model; the quality model is used to determine an approximate threshold that satisfies the quality loss constraint for each layer of the network. The quality model is executed on a GPU and generates configuration information for the approximate threshold. The execution flow of the quality model includes: Sort the activation data for each layer, extract the data values ​​located in the top 20%, top 40%, top 60%, and top 80% of the activation data distribution for that layer, and find the data value that is closest to these data values ​​and can be represented in the form of 2^m + 2^h + 2^k as the approximate threshold for that layer. After completing the selection of the approximate threshold for each layer, traverse all the combinations of approximate thresholds for each layer, and select the N configurations that maximize the network data compression rate from the configurations that satisfy the quality loss constraints.

2. The high real-time approximate on-chip network for accelerating convolutional neural networks as described in claim 1, characterized in that: The shift-multiply-accumulate module consists of three position registers, three weight registers, an adder, and a shifter. The shift-multiply-accumulate module receives the output of the sparse rearrangement module, which is less than the approximate threshold, as input.

3. The high real-time approximate on-chip network for accelerating convolutional neural networks as described in claim 1, characterized in that: The recovery logic includes: After the data packet is transmitted to the destination node, the recovery module extracts the approximate information in the header flit and restores the activation data to the threshold of the left boundary within the segment based on the approximate level of the activation data.

4. The high real-time approximate on-chip network for accelerating convolutional neural networks as described in claim 3, characterized in that: The distribution characteristics of activated data in the network after approximate communication are derived by analyzing the structure and data characteristics of the convolutional neural network.

5. The high real-time approximate on-chip network for accelerating convolutional neural networks as described in claim 1, characterized in that: The approximation module includes a data FIFO, an approximate data FIFO, an approximate threshold register, a comparator, and an approximation level encoder; The recovery module includes a flit decoder, an approximation level register, an approximation threshold register, and a data FIFO; The flit decoder extracts the approximate information from the flit header and stores it in the corresponding position in the approximate level register. The data selector selects different data based on the approximate level information and stores them in the data FIFO.

Images