High-real-time approximate network-on-chip for convolutional neural network acceleration

Abstract

The invention relates to the technical field of approximate network-on-chip, discloses a convolutional neural network acceleration-oriented high-real-time approximate network-on-chip, and provides an approximate compression and segmented recovery scheme based on data classification in combination with the data distribution condition of a neural network, so that the reasoning precision of the neural network is kept while the communication overhead is reduced; a quality control model is provided, and the model can quickly search threshold values of all layers of the neural network meeting quality loss constraints, so that the network is ensured to keep accuracy while performing approximate communication; a threshold value dynamic adjustment scheme based on congestion sensing is further provided, and the approximate threshold value of each layer is dynamically adjusted by sensing the congestion condition in the network, so that the real-time optimization of the data compression rate is realized, and the congestion problem in the communication network is effectively relieved; and in combination with activated data distribution characteristics in a network after approximate communication, an approximate calculation module in the calculation array is optimized, the overall efficiency of the calculation array is improved, and calculation acceleration is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of approximate network-on-chip, and in particular to a high real-time approximate network-on-chip for accelerating convolutional neural networks. Background Art

[0002] Since the birth of Moore's Law, each new generation of process will make the area of ​​a single transistor smaller, thus bringing more transistors in the same area. Even if the computing power on a single computing core continues to increase, due to limitations such as power consumption wall and storage wall, a single core cannot be infinitely large, so multi-core systems on chip came into being. The emergence of multi-core systems on chip solves the problem of insufficient computing power in hardware for high-performance computing applications. These high-performance multi-core processors need to provide a large enough transmission bandwidth while maintaining extremely low transmission latency. Therefore, the network on chip has become an efficient on-chip communication architecture connecting processor cores. The network on chip realizes the interconnection between nodes through routers, and multiple communication data streams are time-division multiplexed on these links, thereby achieving high scalability and large transmission bandwidth.

[0003] Research shows that neural networks have strong fault tolerance, and their fault tolerance makes it possible to use approximate technology during reasoning. As the scale of neural networks increases, the number of parameters increases exponentially, which puts heavy pressure on the communication and computing of multi-core neural network accelerators based on on-chip networks, resulting in a sharp increase in the total execution time, which cannot meet the real-time application scenarios. At present, some research on neural network accelerators based on on-chip networks at home and abroad is mainly focused on the field of approximate communication. These works are based on the constraints on application quality, and reduce the communication load in the on-chip network by lossy compression of data, thereby reducing the average delay of the network to meet the real-time requirements. However, these works did not take into account the nature of the application itself, such as data distribution, and thus did not effectively increase the proportion of compressed data packets; at the same time, they did not coordinate optimization with approximate communication in computing, thus limiting the further reduction of the total execution time.

[0004] Approximate communication is defined as the application of approximate technology in parallel systems, which aims to reduce the amount of communication between PEs. Existing NoC-based approximate communication schemes can be divided into two categories: approximating the entire data in flits and approximating the lowest bits of data in flits. Approximate the entire data in flits often uses a fixed interval to discard the flits at the source node and recover them by linear interpolation at the destination node. Such approximation and recovery schemes limit the total data compression rate to no more than 50%. At the same time, due to the use of fixed intervals for discarding, activation values ​​that have a greater impact on the neural network inference results may be discarded, thereby reducing the model accuracy. Approximate the low bits in the flit data are often implemented by pattern compression and recovery. However, neural network accelerators now often use low-bit quantization. By cutting off the low bits of data, large errors will be introduced. Under the constraints of quality constraints, the truncation bit width is limited, resulting in low data compression rate and inability to alleviate communication congestion.

[0005] When convolutional neural network is inferred, there are a large number of concurrent multiplication and accumulation operations. The power consumption and delay of the multiplication and accumulation unit have become one of the bottlenecks that limit the performance improvement of neural network accelerators. The fault-tolerant characteristics of convolutional neural networks make it possible to use approximate calculations during inference. Under the premise that the quality loss of neural network is acceptable, the power consumption and delay of neural network accelerators can be reduced by introducing approximate units such as approximate multipliers and approximate adders. In the computational power consumption of convolutional neural networks, the power consumption of multipliers accounts for 90% of the computational power consumption. Therefore, introducing approximation for multipliers can effectively reduce the computational power consumption of neural network accelerators and improve the performance of accelerators. The existing approximate multiplier designs are mainly divided into two categories. One category does not change the logic of the multiplier and reduces the power consumption of the multiplier by reducing the voltage. The other category introduces approximation by simplifying some circuit units of the precise multiplier to achieve optimization of power consumption and delay. The operation of reducing the power consumption of the multiplier by voltage reduction will increase the delay of the critical path of the circuit and fail to meet the real-time requirements; while the introduction of approximation by simplifying the circuit logic of the precise multiplier does not take into account the data distribution characteristics of the neural network, which can easily lead to insufficient or excessive approximation. Insufficient approximation does not fully utilize the fault-tolerant characteristics of the neural network, and excessive approximation will lead to excessive quality loss and unacceptable reasoning accuracy. In addition, the current design of approximate multipliers for neural network acceleration has not been organically combined with communication.

[0006] For example, a Chinese patent with publication number CN117411561A discloses a method for network-on-a-chip communication based on approximate computing and a network-on-a-chip. The method for network-on-a-chip communication applies approximate computing technology to the network-on-a-chip and combines power control to achieve reliable low-power communication. Compared with traditional communication, this method reasonably utilizes the inherent robustness of the application, reduces the movement of data during the communication process through approximate computing technology, and sets the required power for communication, thereby avoiding power waste while meeting reliability requirements and improving the power efficiency and execution speed of the network-on-a-chip.

[0007] For example, a Chinese patent with the authorization announcement number CN109547263B discloses a method for optimizing a network on a chip based on approximate computing, which includes a data trimmer, a data restorer, a traffic predictor, a global controller, and a local controller; the data trimmer trims the data before injecting the data packet into the network port to shorten the length of the data packet; the data restorer recovers the lost data after receiving the trimmed data packet; the traffic predictor predicts the data flow in the next control interval based on the past node communication situation; the global controller is used to calculate the optimal configuration of the approximate computing of each node based on the global information and the user's quality requirements from a global perspective, and send the control information to each node; the local controller configures the data loss rate of each data packet waiting to be injected into the network according to the received control information. This method can optimize the performance and power consumption of the network on a chip at a relatively low cost without violating the user's requirements for output quality.

[0008] The above patents have the problems raised by this background technology: the above two patents do not solve the problem of how to effectively compress data using the inter-layer data of the convolutional neural network, nor do they organically combine the approximate computing module of neural network acceleration with communication. To solve this problem, the present invention proposes a high real-time approximate on-chip network for convolutional neural network acceleration. Summary of the invention

[0009] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0010] In view of the above-mentioned problems existing in the existing high real-time approximate on-chip network for convolutional neural network acceleration, the present invention is proposed.

[0011] Therefore, the object of the present invention is to provide a high real-time approximate on-chip network for convolutional neural network acceleration.

[0012] To solve the above technical problems, the present invention provides a high real-time approximate on-chip network for convolutional neural network acceleration: a global threshold dynamic controller in a central node and a local controller in each node;

[0013] The global threshold dynamic controller is used to monitor the congestion in the network and adjust the approximate threshold of each layer node according to the congestion situation;

[0014] The local controller includes an approximation module and a recovery module, the approximation module approximates the data packet according to the global threshold, and the recovery module recovers the data packet according to the approximation information in the header flit of the data packet;

[0015] The approximate multiplication-accumulation unit in each node adopts an approximate calculation module that matches the distribution characteristics in the restored data packet. The approximate calculation module includes a sparse rearrangement module, a shift multiplication-accumulation module and an approximate multiplication-accumulation module.

[0016] As a preferred solution of the high real-time approximate on-chip network for convolutional neural network acceleration described in the present invention, wherein: the sparse rearrangement module is used to perform zero-value sparse processing on the input data to obtain activation data, and rearrange the non-zero data;

[0017] The shift-multiply-accumulate module recognizes the recovered non-zero data and implements multiplication by shifting;

[0018] The approximate multiplication and accumulation module calculates the activation data that has not been processed approximately during the approximate communication process;

[0019] The results of the shift multiply-accumulate module and the approximate multiply-accumulate module are reduced, and the data is output for transmission to the next stage.

[0020] As a preferred solution of the high real-time approximate on-chip network for convolutional neural network acceleration described in the present invention, wherein: the global threshold controller is used to implement a dynamic threshold adjustment solution based on congestion perception;

[0021] The threshold dynamic adjustment scheme based on congestion perception is to dynamically adjust the approximate threshold of each layer of nodes by sensing the congestion situation in the network;

[0022] The threshold dynamic adjustment scheme based on congestion perception includes:

[0023] The number of each layer of nodes is initialized by the global threshold controller, and the network layer numbers corresponding to different nodes and the number of nodes in each layer of the network are recorded. The global threshold controller obtains the congestion situation of each layer of nodes. The congestion situation of each layer of nodes is measured by the free slot occupancy rate of the local buffer. If congestion occurs in a node of a certain layer, the congest flag of the node is set to 1, and the number of nodes in congestion in each layer is counted. The global threshold controller calculates the ratio of the number of nodes in congestion in each layer to the total number of nodes in the layer to evaluate which layer is the most congested. If the approximate thresholds configured for the most congested layer are the same, the secondary congested layer is found, and the selected configuration information is determined by the approximate thresholds of different configurations of the secondary congested layer.

[0024] As a preferred solution of the high real-time approximate on-chip network for convolutional neural network acceleration described in the present invention, the shift-multiply-accumulate module is composed of three position registers, three weight registers, an adder and a shifter, and the shift-multiply-accumulate module receives the output less than the approximate threshold from the sparse rearrangement module as input.

[0025] As a preferred solution of the high real-time approximate on-chip network for convolutional neural network acceleration described in the present invention, the approximate communication includes data hierarchical compression and recovery, and the logic of the data hierarchical compression includes:

[0026] Set the approximate thresholds T0~Tn. If the activation data of the layer is less than T0, the approximate level of the activation data of the layer is set to 0;

[0027] If the activation data of the layer is between T0 and T1, the approximation level of the activation data of the layer is set to 1, and so on. For the activation data greater than Tn-1 but less than Tn, the approximation level of the activation data of the layer is set to n-1;

[0028] The position and approximation level of the activation data to be approximated are recorded in the header flit, and the activation data to be approximated are deleted from the data packet. The activation data greater than Tn is not approximated.

[0029] As a preferred solution of the high real-time approximate network-on-chip for convolutional neural network acceleration described in the present invention, the recovery logic includes:

[0030] After the data packet is transmitted to the destination node, the recovery module extracts the approximation information in the header flit, and restores the activation data to the threshold of the left boundary in the segment according to the approximation level of the activation data.

[0031] As a preferred solution of the high real-time approximate on-chip network for convolutional neural network acceleration described in the present invention, wherein: the approximate threshold is obtained through a quality model;

[0032] The quality model is used to determine an approximate threshold that satisfies a quality loss constraint for each layer of the network, and the quality model is executed on a GPU and generates configuration information of the approximate threshold;

[0033] The execution process of the quality model includes:

[0034] Sort the activation data of each layer, extract the data values ​​located in the top 20%, top 40%, top 60% and top 80% of the activation data distribution of this layer, find the nearest data value of these data values ​​that can be expressed in the form of 2^m+2^n+2^k as the approximate threshold of this layer, and after completing the selection of the approximate threshold of each layer, traverse all approximate threshold combinations of each layer, and select N configurations that maximize the network data compression rate among the configurations that meet the quality loss constraint.

[0035] As a preferred solution for the high real-time approximate on-chip network for accelerating convolutional neural networks described in the present invention, the distribution characteristics of the activation data in the network after the approximate communication are obtained by analyzing the structure and data characteristics of the convolutional neural network.

[0036] As a preferred solution of the high real-time approximate on-chip network for convolutional neural network acceleration described in the present invention, wherein: the approximate module includes a data FIFO, an approximate data FIFO, an approximate threshold register, a comparator and an approximate level encoder;

[0037] The recovery module includes a flit decoder, an approximation level register, an approximation threshold register and a data FIFO;

[0038] The flit decoder extracts the approximation information in the header flit and stores it in the corresponding position in the approximation level register. The data selector selects different data according to the approximation level information and stores it in the data FIFO.

[0039] Beneficial effects of the present invention: The present invention proposes an approximate compression and segmented recovery scheme based on data classification in combination with the data distribution in the neural network. By compressing the data in a hierarchical manner, the accuracy of neural network reasoning is maximized while reducing the communication overhead; and a quality control model is proposed, which can quickly search for the thresholds of each layer of the neural network that meet the quality loss constraint, ensuring that the network maintains accuracy while performing approximate communication; on the basis of the quality control model, a threshold dynamic adjustment scheme based on congestion perception is further proposed, which dynamically adjusts the approximate threshold of each layer by sensing the congestion in the network to achieve real-time optimization of the data compression rate, thereby effectively alleviating the congestion problem in the communication network; combined with the distribution characteristics of the activated data in the network after approximate communication, the approximate computing module in the computing array is optimized, the overall efficiency of the computing array is improved, and computing acceleration is achieved. Through this method, the total execution time of neural network reasoning is greatly shortened, and the real-time performance and performance of the system are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0041] Figure 1 This is a general architecture diagram of a fused approximate network-on-chip of a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0042] Figure 2 This is a distribution diagram of AlexNet activation data of a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0043] Figure 3 It is a distribution diagram of activation data of a certain layer after approximate communication of AlexNet, a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0044] Figure 4 A sparse rearrangement module diagram of a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0045] Figure 5 A shift-multiply-accumulate module diagram of a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0046] Figure 6 This is an approximate multiplication-accumulation module diagram of a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0047] Figure 7 An approximate module diagram of a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0048] Figure 8 A recovery module diagram of a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0049] Fig. 9 A total execution time comparison diagram of a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0050] Fig.10 A data compression rate comparison diagram of a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0051] Fig.11 This is a comparison chart of the computing time of a high real-time approximate network-on-chip for convolutional neural network acceleration according to the present invention;

[0052] Fig.12 This is a communication time comparison diagram of a high real-time approximate on-chip network for convolutional neural network acceleration according to the present invention. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0055] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0056] Example

[0057] like Figure 1 As shown, a high real-time approximate on-chip network for convolutional neural network acceleration includes a global threshold dynamic controller in a central node and a local controller in each node.

[0058] The global threshold dynamic controller is used to monitor the congestion in the network and adjust the approximate threshold of each layer node according to the congestion situation;

[0059] The local controller includes an approximation module and a recovery module, the approximation module approximates the data packet according to the global threshold, and the recovery module recovers the data packet according to the approximation information in the header flit of the data packet;

[0060] The approximate multiplication and accumulation unit in each node adopts an approximate calculation module that matches the distribution characteristics in the restored data packet. The approximate calculation module includes a sparse rearrangement module, a shift multiplication and accumulation module, and an approximate multiplication and accumulation module.

[0061] The sparse rearrangement module pre-fetches multiple valid activations, then detects the positions where the activations are zero and cleans the data; the shift multiplication-accumulation module recognizes the recovered non-zero values ​​and implements multiplication by shifting; the approximate multiplication-accumulation module uses the improved Drum multiplier to calculate the activation data that has not been approximated during the approximate communication process; finally, the results of the two multiplication-accumulation modules are merged into the final result for transmission to the next level.

[0062] Existing research shows that in different convolutional neural network structures and different data sets, the weight data and activation data of each layer of the convolutional neural network mostly show the characteristics of Gaussian distribution or truncated Gaussian distribution. This distribution characteristic is mainly due to the addition of a normalization layer in each layer of the neural network, which adjusts the distribution of activation data and weight data of each layer to be symmetrical about the zero axis.

[0063] At the same time, the convolutional layer is usually followed by a ReLU activation function, which truncates all negative values ​​to zero, causing the activation data to exhibit the characteristics of a truncated Gaussian distribution and resulting in a large number of zero values, making the output data highly sparse. Inspired by the principle of neural network pruning, studies have shown that when weights or activation values ​​close to zero are directly set to zero, the impact on the network's reasoning accuracy is very limited.

[0064] Therefore, we proposed a hierarchical compression and segmented recovery scheme based on the amplitude of activation data. By compressing the activation data at different levels according to the amplitude, we can achieve efficient data compression, and through segmented recovery, we can minimize the communication overhead while ensuring network accuracy.

[0065] Before introducing the principle of the data compression and recovery scheme, we first need to introduce the composition of data packets when communicating between processing units (PEs) in the network on chip (NoC). A data packet usually consists of three parts: head flit, body flit and tail flit. The head flit contains the key information of the data packet, including the type tag of the flit, the source node coordinates, the destination node coordinates, the virtual channel sequence number, the routing algorithm and the neuron index, etc., which are used to guide the transmission and processing of the data packet; the body flit carries the main data load, including the type, sequence number and specific data information of the flit; the tail flit can be empty, or consistent with the data information in the body flit, which is used to mark the end of the data packet.

[0066] Usually, a complete data packet consists of a header flit and a tail flit, with N individual flits in the middle, which carry the actual data information. The information of the header flit is extended and designed to record the location of the approximate data in the data packet and the approximate level of the data. This improvement enables the destination node to effectively recover the received approximate data based on this additional information.

[0067] Taking the activation output of a certain layer of the neural network as an example, a specific scheme based on hierarchical data compression and segmented recovery is proposed. The principle of this scheme is: first, a series of approximate thresholds T0~Tn are set. For the activation data of this layer, if the activation data is less than T0, its approximation level is set to 0; if the activation data is between T0 and T1, its approximation level is set to 1, and so on. For activation data greater than Tn-1 but less than Tn, its approximation level is set to n-1.

[0068] The position of the approximated body flit and the corresponding approximation level will be recorded in the header flit, and the approximated activation data will be deleted from the data packet, and the subsequent flits will be moved forward, thereby shortening the length of the data packet and improving transmission efficiency; for activation data with an amplitude exceeding the approximation threshold interval, that is, activation data greater than Tn, no approximation will be performed and its original accuracy will be retained.

[0069] After the data packet is transmitted to the destination node, the recovery module extracts the approximate information in the header flit and performs segmented data recovery based on this approximate information. The specific recovery process is to restore the activation data to the threshold of the left boundary within the segment based on the approximation level. For example, the activation data with an approximation level of 0 will be restored to 0, the activation data with an approximation level of 1 will be restored to T0, and so on, thereby completing the reconstruction of the approximate data. In this process, the balance between the compression efficiency and recovery accuracy of the activation data is guaranteed, maximizing the compression rate while minimizing the impact on the network reasoning accuracy.

[0070] In order to achieve the above-mentioned approximate communication based on data hierarchical compression and recovery scheme, a quality model is proposed to quickly determine the approximate threshold that satisfies the quality loss constraint for each layer of the network. This process is executed on the GPU and generates a threshold information called a "configuration table".

[0071] The specific process is as follows: first, a complete inference process is executed, and then the activation data of each layer is sorted, and the data values ​​​​in the top 20%, top 40%, top 60%, and top 80% of the activation data distribution of this layer are extracted. The reason for choosing these proportions is that experimental results show that under the data hierarchical compression and recovery scheme, a certain layer of most neural networks can approximately process up to 80% of the data. The four levels facilitate congestion perception and dynamic regulation during network operation.

[0072] Next, find the data value closest to these data values ​​that can be expressed in the form of 2^m+2^n+2^k as the approximation threshold of this layer. Selecting these approximation thresholds helps speed up calculations and improve energy efficiency. The quality loss of a neural network is closely related to the approximation rate of the data. When a layer is congested, choosing a larger approximation threshold means that the data compression rate (i.e., approximation rate) of this layer will be higher. However, in order to meet the overall quality loss constraint, it is necessary to reduce the data approximation rate of other layers, that is, reduce the approximation thresholds of these layers.

[0073] Then, after completing the selection of the approximate threshold for each layer, considering that not all threshold combinations can meet the quality loss constraint, it is necessary to traverse all possible approximate threshold combinations to find the optimal solution. In order to reduce the computational overhead caused by this traversal, the search space of the approximate threshold is reduced by merging the network layers with fewer occupied nodes. For example, when a network has 19 layers, it is theoretically necessary to traverse 4^18 possible combinations (the last layer is not approximated, so only 18 layers need to select the approximate threshold), which will result in a huge time overhead. However, after merging the layers with fewer occupied nodes, the search space will be significantly reduced. In addition, this layer binding operation can make the number of nodes adjusted each time more uniform during the dynamic control process, thereby making the network load distribution more balanced when reasoning on the NoC.

[0074] In specific applications, the neural network quality loss threshold is set to 2%. After traversing to obtain the approximate threshold configuration, the approximate communication and approximate calculation are further combined, the neural network is inferred on the C++ platform, and the approximate multiplication and accumulation unit proposed in the invention is used to replace the precise approximate multiplication and accumulation unit for reasoning, and num threshold configurations that meet the quality loss constraint under the combination of calculation and communication are saved.

[0075] When a neural network is deployed on a NoC for inference, the traffic of each layer of the network will change dynamically over time. In order to effectively alleviate communication congestion, a congestion-aware threshold dynamic adjustment scheme is proposed. This scheme embeds a global dynamic threshold controller in the global control node to sense the congestion of each layer of nodes in real time, and dynamically adjusts the approximate threshold of each layer of nodes according to these conditions, thereby alleviating congestion problems that may occur during system operation.

[0076] The controller will first initialize the number of each node, and record the network layer number corresponding to different nodes and the number of nodes in each layer of the network; then, the controller will obtain the congestion status of each local node. The congestion of the local node is measured by the free slot occupancy rate of the local buffer. If a node is congested, the system will set the congest flag of the node to 1 and count the number of congested nodes in each layer. For example, there are 3 congested nodes in the first layer and 2 congested nodes in the second layer; then, the controller calculates the ratio of the number of congested nodes in each layer to the total number of nodes in that layer, so as to evaluate which layer is the most congested; then use the greedy algorithm, if there are multiple configuration thresholds of the most congested layer with the same threshold, then find the second most congested layer, and determine the final selected configuration information through the thresholds of different configurations of this layer, and so on.

[0077] This congestion-aware dynamic threshold adjustment scheme can adjust the approximate threshold of each layer in real time during network operation according to the relative congestion of different layers, significantly alleviating the congestion problem in different applications and mapping scenarios, thereby improving the system's operating speed and shortening the total execution time of neural network reasoning. Once the optimized threshold configuration is obtained, the global control node will send it to each local node. After receiving the new approximate threshold, the approximation module and recovery module in the local node will perform hierarchical approximation and segmented recovery operations of the data according to the updated threshold information to ensure the efficiency of network operation.

[0078] Traditional approximate neural network accelerators usually only consider one aspect of approximate communication or approximate computing, and fail to organically combine the two, thus failing to further reduce the total execution time of neural network reasoning. In view of this limitation, a low-latency and low-power approximate computing module is designed by utilizing the data distribution characteristics after approximate communication, which can be organically combined with approximate communication to achieve computing acceleration. As mentioned above, the inter-layer activation data of convolutional neural networks usually presents a Gaussian-like distribution or a truncated Gaussian-like distribution (such as Figure 2 As shown in Figure 2), after being processed by the proposed hierarchical compression and segmented recovery scheme, the activation data restored by the approximate communication at the destination node shows the characteristics of discrete distribution below the threshold and continuous distribution above the threshold (as shown in Figure 2). Figure 3 shown).

[0079] In addition, since the proportion of 0 values ​​in the restored activation data increases significantly, based on this feature, the approximate computing module proposed in the present invention consists of three core sub-components: a sparse rearrangement module, a shift multiplication and accumulation module, and an approximate multiplication and accumulation module.

[0080] The main function of the sparse rearrangement module is to perform zero-value sparse processing on the input data and rearrange the non-zero data to improve the computing efficiency; the shift multiplication and accumulation module is responsible for processing the data that has been approximately processed in the approximate communication, and the approximate multiplication and accumulation module uses the improved Drum multiplier to calculate the data that has not been approximately processed in the approximate communication process to further reduce the computing power consumption; the sparse rearrangement module (such as Figure 4 As shown in FIG. 1 , a batch of activation data is pre-fetched in one cycle, zero values ​​are filtered out through a comparator, and then the approximately restored data and the non-approximately processed data are output to the corresponding approximate calculation module through two paths to complete the calculation.

[0081] Shift multiply accumulate module (such as Figure 5 The module (as shown) consists of three position registers, three weight registers, an adder and a shifter. When working, the module receives the output less than the approximate threshold from the sparse rearrangement module as input. First, the specific bit of the input is judged. Since the set approximate threshold conforms to the form of 2^m+2^n+2^k, it is only necessary to judge whether the m, n, and k bits are 1; if they are 1, the corresponding weight register will accumulate the weight corresponding to the data. When the shift-multiply-accumulate module receives the completion signal sent by the data sparse rearrangement module, the values ​​in the three weight registers will be shifted and accumulated, and the final output result.

[0082] For the activation data that has not been approximated, these numbers are often large. In binary multiplication operations, the high bits of the input data have a decisive influence on the accuracy of the final calculation result. Their importance far exceeds the low bits of the data. Therefore, the approximate multiplication and accumulation module (such as Figure 6 As shown in the figure, this module only sends the high K bits of the activation data including the first 1 to the approximate calculation module. It is found through experiments that for most networks, the minimum value of the threshold of each layer is often greater than 256. Therefore, when the approximate multiplication and accumulation module detects the first 1 of the activation value, it only needs to detect the high 8 bits. When K is fixed to 6, the loss of inference accuracy is within 2% when the approximate on-chip network is used to accelerate most convolutional neural networks. At the same time, the approximate calculation module can not only skip the 0-value operation, but also calculate the approximate value and non-approximate value in the approximate communication process in two parallel ways, and achieves significant calculation acceleration with a small additional overhead.

[0083] Figure 7It is the approximation module in the local node, which includes data FIFO, approximate data FIFO, approximate threshold register, comparator, approximate level encoder, etc. First, we cache the data in the storage module (Buffer) through the data FIFO. As long as the FIFO is not empty, we read the FIFO and compare the read data with the four thresholds in the approximate threshold register; then encode (combine) the comparison result into the approximate level, and the encoded approximate level is stored in the approximate level register. At the same time, we control the data selector after bitwise OR of the approximate level, which can discard the approximate data or save it in the approximate data FIFO; the data in the body flit that is not approximated is packaged by the valid data in the approximate data FIFO.

[0084] Figure 8 It is the recovery module in the local node. This module includes a flit decoder, an approximation level register, an approximation threshold register and a data FIFO. The flit decoder extracts the approximation information in the header flit and stores it in the corresponding position in the approximation level register; then the data selector selects different data according to the approximation level information and stores it in the data FIFO.

[0085] Beneficial effects of the present invention in actual simulation:

[0086] In order to verify the effect of the proposed architecture in shortening the total execution time of neural network accelerator inference, simulation was carried out on the CNN-Noxim platform. We set the NoC to an 8*8 2D-mesh structure, each data packet contains 8 flits, each flit is 36 bits, the mapping of the neural network on the NoC adopts random mapping, and the routing algorithm is XY routing; the benchmark used is the convolutional neural network based on CIFAR10, including AlexNet, VGG11, VGG13, VGG16, ResNet8, all networks use 16bit quantization, and the accuracy loss does not exceed 2%; compared with the most advanced approximate communication frameworks including ACDC_ABDTR, ACDC_APPROX, Neu_NoC, and also compared with the neural network accelerator of sparse computing, the proposed framework is FAS-NoC.

[0087] The comparison dimensions include total execution time and data compression ratio (approximation ratio) (such as Fig. 9 and Fig.10As shown). The total execution time of different architectures is normalized for NO_APPROX which does not use approximation technology; the higher the data compression rate, the more flits are discarded during the approximate communication process, and the lower the communication delay. From the results, it can be seen that the proposed FAS-NoC has a significant effect on reducing the total execution time and improving the data compression rate. This is because the approximation method of APPROX_NoC is truncated approximation, and the pattern matching mechanism is used to compress high-bit data, while 16-bit quantization weights and activations are used when performing neural network inference, which results in a smaller bit width that can be truncated when the data accuracy loss is less than 2%, and at the same time, the compression of high-bit data is limited, and the data approximation rate is low.

[0088] The ACDC_ABDTR scheme does not monitor the value of the transmitted data and discards flits at fixed intervals, which may cause some critical large values ​​to be discarded. Under the constraint of 2% quality loss, the number of packets that can be discarded is extremely limited. The sparse computing scheme does not consider the optimization of communication, resulting in limited optimization of its execution time.

[0089] In addition, in order to demonstrate the optimization of the invention in communication delay and computing delay, the communication and computing time of the first few convolutional layers of VGG16 were counted respectively. Fig.11 and Fig.12 This result shows that the proposed FAS-NoC not only alleviates communication delay, but also achieves computing acceleration. Compared with the sparse computing accelerator, the computing delay is further reduced. This is because a part of the activations in 0-T0 are approximated to 0, which further increases the sparsity of the network and saves more computing time.

[0090] The present invention organically combines approximate communication and approximate computing to significantly reduce the total execution time of neural network reasoning.

[0091] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A high real-time approximate network-on-chip for convolutional neural network acceleration, characterized by: include: a global threshold dynamic controller at the central node and local controllers in each node; The global threshold dynamic controller is used to monitor the congestion in the network and adjust the approximate threshold of each layer node according to the congestion situation; The local controller includes an approximation module and a recovery module, the approximation module approximates the data packet according to the global threshold, and the recovery module recovers the data packet according to the approximation information in the header flit of the data packet; The approximate multiplication-accumulation unit in each node adopts an approximate calculation module that matches the distribution characteristics in the restored data packet. The approximate calculation module includes a sparse rearrangement module, a shift multiplication-accumulation module and an approximate multiplication-accumulation module.

2. A high real-time approximate network-on-chip for convolutional neural network acceleration as claimed in claim 1, characterized in that: The sparse rearrangement module is used to perform zero-value sparse processing on the input data to obtain activation data and rearrange the non-zero data; The shift-multiply-accumulate module recognizes the recovered non-zero data and implements multiplication by shifting; The approximate multiplication and accumulation module calculates the activation data that has not been processed approximately during the approximate communication process; The results of the shift multiply-accumulate module and the approximate multiply-accumulate module are reduced, and the data is output for transmission to the next stage.

3. A high real-time approximate network-on-chip for convolutional neural network acceleration as claimed in claim 1, characterized in that: The global threshold controller is used to implement a dynamic threshold adjustment solution based on congestion perception; The threshold dynamic adjustment scheme based on congestion perception is to dynamically adjust the approximate threshold of each layer of nodes by sensing the congestion situation in the network; The threshold dynamic adjustment scheme based on congestion perception includes: The number of each layer of nodes is initialized by the global threshold controller, and the network layer numbers corresponding to different nodes and the number of nodes in each layer of the network are recorded. The global threshold controller obtains the congestion situation of each layer of nodes. The congestion situation of each layer of nodes is measured by the free slot occupancy rate of the local buffer. If congestion occurs in a node of a certain layer, the congest flag of the node is set to 1, and the number of nodes in congestion in each layer is counted. The global threshold controller calculates the ratio of the number of nodes in congestion in each layer to the total number of nodes in the layer to evaluate which layer is the most congested. If the approximate thresholds configured for the most congested layer are the same, the secondary congested layer is found, and the selected configuration information is determined by the approximate thresholds of different configurations of the secondary congested layer.

4. A high real-time approximate network-on-chip for convolutional neural network acceleration as claimed in claim 2, characterized in that: The shift-multiply-accumulate module is composed of three position registers, three weight registers, an adder and a shifter. The shift-multiply-accumulate module receives the output from the sparse rearrangement module that is less than the approximate threshold as input.

5. A high real-time approximate network-on-chip for convolutional neural network acceleration as claimed in claim 2, characterized in that: The approximate communication includes data hierarchical compression and recovery, and the logic of the data hierarchical compression includes: Set the approximate thresholds T0~Tn. If the activation data of the layer is less than T0, the approximate level of the activation data of the layer is set to 0; If the activation data of the layer is between T0 and T1, the approximation level of the activation data of the layer is set to 1, and so on. For the activation data greater than Tn-1 but less than Tn, the approximation level of the activation data of the layer is set to n-1; The position and approximation level of the activation data to be approximated are recorded in the header flit, approximation information is generated, and the activation data to be approximated is deleted from the data packet. The activation data greater than Tn is not approximated.

6. A high real-time approximate network-on-chip for convolutional neural network acceleration as claimed in claim 5, characterized in that: The recovery logic includes: After the data packet is transmitted to the destination node, the recovery module extracts the approximation information in the header flit, and restores the activation data to the threshold of the left boundary in the segment according to the approximation level of the activation data.

7. A high real-time approximate network-on-chip for convolutional neural network acceleration as claimed in claim 6, characterized in that: The approximate threshold is obtained through a quality model; The quality model is used to determine an approximate threshold that satisfies a quality loss constraint for each layer of the network, and the quality model is executed on a GPU and generates configuration information of the approximate threshold; The execution process of the quality model includes: Sort the activation data of each layer, extract the data values ​​located in the top 20%, top 40%, top 60% and top 80% of the activation data distribution of this layer, find the data value closest to these data values ​​that can be expressed in the form of 2^m+2^n+2^k as the approximate threshold of this layer, and after completing the selection of the approximate threshold of each layer, traverse all approximate threshold combinations of each layer, and select N configurations that maximize the network data compression rate among the configurations that meet the quality loss constraint.

8. A high real-time approximate network-on-chip for convolutional neural network acceleration as claimed in claim 7, characterized in that: The activation data distribution characteristics in the network after the approximate communication are obtained by analyzing the structure and data characteristics of the convolutional neural network.

9. The high real-time approximate network-on-chip for convolutional neural network acceleration according to claim 1, characterized in that: The approximation module includes a data FIFO, an approximation data FIFO, an approximation threshold register, a comparator and an approximation level encoder; The recovery module includes a flit decoder, a proximity level register, a proximity threshold register and a data FIFO; The flit decoder extracts the approximation information in the header flit and stores it in the corresponding position in the approximation level register. The data selector selects different data according to the approximation level information and stores it in the data FIFO.

Images