Sparse perception scheduler, hybrid acceleration architecture, intelligent edge chip and equipment
Through sparse perception scheduler and hybrid acceleration architecture, the neural network load is dynamically scheduled, which solves the demand for hardware resources of complex neural network models in edge computing, realizes the training process of low latency and high resource utilization, and improves the training of neural networks Efficiency and performance.
Patent Information
- Application Number
- CN202410813110.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-23
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-06-23
AI Technical Summary
In edge computing, the demand for hardware resources by complex neural network models leads to latency and resource utilization problems, especially under the limited hardware resources of edge devices, it is difficult to achieve efficient parallel computing and low-latency training processes.
Adopting sparse perception scheduler and hybrid acceleration architecture, the neural network load is dynamically scheduled through sparse information extractor, delay estimation unit, load distributor and forward detector, and using the CPU's pipeline acceleration function, parallel computing, and FP and BP calculations are arranged. To different acceleration units, optimize latency and resource utilization according to sparseness levels.
Low latency time and high resource utilization are achieved, which significantly improves the efficiency and performance of neural network training, reduces time gaps, and improves the utilization of hardware accelerators.
Smart Images

Figure CN118921302A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of integrated circuit technology, and in particular to a sparse-aware scheduler, a hybrid acceleration architecture, an intelligent edge chip and a device. Background Art
[0002] Edge computing is a type of distributed computing that migrates computing tasks and data storage from data centers or cloud environments to the edge of the network, that is, closer to the data source. In edge computing, data does not need to be uploaded directly to the cloud or centralized data processing system, which not only reduces the latency of data transmission and improves the response speed of services, but also reduces the computing and storage burden on the cloud, improves data privacy and security, and reduces the risk of data leakage. In addition, edge computing can also save network bandwidth because it reduces the transmission of large amounts of data in the network, which is particularly suitable for applications that process large amounts of data, such as Internet of Things (IoT) devices, autonomous driving, etc. More importantly, edge computing can support real-time or near real-time data processing, which is of great value for applications that require rapid decision-making (such as autonomous driving, industrial automation, etc.).
[0003] An intelligent edge chip is a microprocessor designed specifically for edge computing. Its powerful computing power is packaged in a small hardware device to meet the edge computing requirements for low latency, high efficiency, and real-time processing. With the development of artificial intelligence and machine learning, some intelligent edge chips have begun to integrate specialized AI accelerators, such as neural network processors (NPUs). This type of chip can perform inference of deep learning models on the device side, greatly improving processing efficiency and response speed while also protecting data privacy. As the core of edge devices, intelligent edge chips have stronger processing and computing capabilities and can perform artificial intelligence calculations. Neural networks have now deeply penetrated into all aspects of people's lives. As people's understanding and application of neural networks continue to deepen, current neural network models are becoming more and more complex, which undoubtedly increases the requirements for hardware performance.
[0004] As neural network models add more layers, more nodes, and more complex connections, the amount of data they need to process also increases, which requires hardware devices to have more powerful computing capabilities and larger memory capacity. As a result, the growing number of neural network model parameters and the limited hardware resources of edge devices are increasingly in conflict. In addition, in order to train these complex neural network models, hardware devices also need to support efficient parallel computing and high-speed data transmission to ensure that they can maintain efficient operating speeds and low-latency response times when processing large amounts of data and complex calculations. Therefore, it has become a demand to provide intelligent edge chips with low latency and high resource utilization. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides a sparse-aware scheduler, a hybrid acceleration architecture, an intelligent edge chip and a device, which have lower latency and higher resource utilization.
[0006] The present invention provides a sparse-aware scheduler, which is applicable to an intelligent edge chip containing multiple accelerators, and includes: a sparse information extractor, a delay estimation unit, a load distributor, and a forwarding detector.
[0007] The sparse information extractor is used to associate a network layer in a neural network with the sparse feature information of the network layer;
[0008] The delay estimation unit is used to evaluate the delay of each network layer when the network layer runs on different accelerators according to the sparse feature information of the network layer;
[0009] The load distributor is used to determine a configuration scheme of the accelerators to obtain a smaller delay based on a forward propagation delay in the first accelerator, a reverse propagation delay in the second accelerator, a backward propagation delay in the first accelerator, and a forward propagation delay in the second accelerator;
[0010] The forwarding detector is used to determine whether the output of the first accelerator and the output of the second accelerator are directly sent to the input buffer area of the other party according to the output result of the load distributor.
[0011] The present invention also provides a hybrid acceleration architecture, comprising: a plurality of accelerators and the above-mentioned sparse-aware scheduler, wherein the plurality of accelerators are used to support sparse training of simultaneous forward propagation and backward propagation.
[0012] The present invention also provides an intelligent edge chip, comprising the above-mentioned hybrid acceleration architecture.
[0013] The present invention also provides an edge device, comprising the above-mentioned intelligent edge chip.
[0014] In the embodiment of the present invention, the IPA unit and the ACC (accelerator) form hybrid acceleration to achieve parallel computing and utilize the pipeline acceleration function of the CPU; at the same time, the sparse-aware scheduler can estimate the delay based on the sparsity level and arrange the FP and BP calculations to different acceleration units. In the scheduler of the present invention, the inter-layer delay is estimated based on the change in the sparsity ratio between different layers, and tasks are dynamically allocated to the two accelerators at the layer granularity, achieving better performance and higher resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0016] Figure 1 A schematic diagram of a hybrid acceleration architecture provided by an embodiment of the present invention;
[0017] Figure 2 for Figure 1 Schematic diagram of the structure of the sparse-aware scheduler in;
[0018] Figure 3 for Figure 1 Schematic diagram of unified memory mapping and read / write mode of unified memory system;
[0019] Figure 4 A schematic diagram comparing the timeline of a traditional architecture, the timeline of sparse training without scheduling, and the timeline of sparse training with scheduling of the present invention;
[0020] Figure 5 The figure shows the time gap comparison and hardware utilization comparison with and without scheduling. DETAILED DESCRIPTION
[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0022] As mentioned above, deep neural networks (DNNs) are widely used in various fields. With the continuous advancement of the network, the number of parameters and the amount of computation have been increasing. Edge devices, such as Internet of Things (IoT) devices, are also equipped with the ability to run deep learning neural networks. These deep learning neural networks specifically for edge applications are called TinyML (TinyMachine Learning). TinyML collects personal data collected by edge devices and runs neural networks locally on the device, avoiding time-consuming and power-consuming cloud data transmission, and can achieve efficient and low-latency computing. In order to realize the deployment of neural networks on the edge, it is necessary to solve the contradiction between the increasing number of model parameters and the limited hardware resources of edge devices from the software and hardware levels. In addition to inference, edge device training is becoming increasingly important due to its unique advantages. Compared with inference, edge device training brings the possibility of neural network personalization. Local training can avoid data being transmitted to the cloud, protect personal privacy data, and customize the training of neural networks deployed on edge devices according to user behavior. By collecting user data to fine-tune the neural network, the prediction accuracy of the personalized network can be improved by continuously fitting user habits. However, the limited hardware resources of edge devices will become more strained during training. The training process includes forward propagation (FP) and backward propagation (BP). Backward propagation includes the calculation of weight gradients and intermediate activation layer gradients, so additional storage space is required to store the results of the intermediate activation layer and the gradients, which occupies 54% of the storage for back propagation. In addition, there is a transposition operation on the matrix in back propagation, which will lead to irregular access patterns to the storage medium.
[0023] In order to alleviate the demand for large storage, many technologies have been introduced into the training process. Sparse training is one of the solutions, which reduces the storage requirement by skipping some bias and weight updates during back propagation, while the loss of accuracy is almost negligible. However, when the sparsity ratio changes, the number of operations required by the network shows a large range of variation, resulting in unpredictable delays for a complete training iteration in practical applications, and the large difference in sparsity ratios between forward and backward propagation is likely to lead to low utilization of hardware accelerators. To this end, the present invention proposes a sparse-aware scheduler in an architecture for hybrid acceleration, which can schedule the neural network load according to the real-time perceived sparsity ratio, which not only reduces the delay time, but also helps to improve the utilization of hardware resources.
[0024] To make the technical solution of the present invention clearer, embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0025] Figure 1 A schematic diagram of a hybrid acceleration architecture provided by an embodiment of the present invention is based on the open source lightweight TinyML platform, which includes a 5-stage pipeline RISC-V CPU and peripheral IO. In order to better support sparse training of forward and backward transmission at the same time, two acceleration units are designed in the architecture, such as Figure 1 As shown, the architecture in this embodiment includes: two accelerators and a sparse-aware scheduler, two accelerators, for supporting sparse training of simultaneous forward propagation and backward propagation. Among them, the first accelerator is a pipeline tightly coupled acceleration unit (IPA), which is used to perform vector multiplication and accumulation, and the second accelerator is a dedicated multiplication and accumulation array (ACC). Through the bus interface of the CPU MEM (memory) stage, the ACC receives a control signal from the CPU, and the green dotted line in the figure indicates the data flow on the system bus. Among them, the pipeline tightly coupled acceleration unit (pipeline tightly coupled accelerator) is configured in the memory access pipeline stage of the central processing unit (CPU) in the intelligent edge chip. The pipeline tightly coupled accelerator (in-pipeline acceleration) achieves considerable performance improvement by designing an accelerator in the pipeline of the CPU with a relatively small area overhead, which is critical for resource-constrained TinyML devices. The pipeline tightly coupled acceleration unit implements a multiplication and accumulation unit, which can perform vector multiplication and accumulation. The present invention places the IPA in the memory access pipeline stage of the CPU, so that the data cache and ACC can be accessed more quickly. IPA will take multiple cycles to complete the propagation of a specified layer, depending on the workload. Input, weight, and output caches are implemented using FIFOs. For each computation cycle, each FIFO will give a set of data to the vector computation unit to complete the vector multiplication operation. The vector computation unit consists of 4 multipliers and an adder tree.
[0026] In order to adapt to more sparse strategies and realize simultaneous forward and reverse propagation, the present invention designs a sparse-aware scheduler in the execution phase of the CPU, which will dynamically schedule forward propagation and reverse propagation according to real-time sparse information, and distribute the workload to IPA and ACC. The sparse-aware scheduler will balance the amount of computation as much as possible to improve hardware utilization and bandwidth. If you want to switch the workload in IPA and ACC, the traditional approach is for IPA / ACC to output to their respective output caches and then read each other. In order to avoid this data transfer overhead, the present invention allows data to be directly transferred between IPA and ACC, similar to data forwarding. The sparse-aware scheduler will send a data forwarding signal, which means that the outputs of the two accelerators will be sent directly to each other. Figure 1The blue dotted line in shows the flow of workload initialization and data forwarding signals. The sparse updater in the write-back phase will receive information submitted by the two accelerators at the end of their operation, after which the sparse ratio information stored in the scheduler will be updated. The sparse-aware scheduler designed by the present invention is described in detail below.
[0027] Figure 2 for Figure 1 The structural diagram of the sparse-aware scheduler in Figure 2 As shown, the sparse-aware scheduler in an embodiment of the present invention includes: a sparse information extractor, a delay estimation unit, a load distributor and a forward detector, wherein the sparse information extractor is used to associate a network layer in a neural network with sparse feature information of the network layer, wherein the sparse feature information includes: a sparse ratio, a sparse pattern and the dimension of a weight matrix; the delay estimation unit is used to evaluate the delay of each network layer when the network layer runs on different accelerators according to the sparse feature information of the network layer, and obtain the forward propagation delay in the first accelerator, the reverse propagation delay in the second accelerator, the backward propagation delay in the first accelerator and the forward propagation delay in the second accelerator; the load distributor is used to determine the configuration scheme of the accelerator according to the forward propagation delay in the first accelerator, the reverse propagation delay in the second accelerator, the backward propagation delay in the first accelerator and the forward propagation delay in the second accelerator to obtain a smaller delay; the forward detector is used to determine whether the output of the first accelerator and the output of the second accelerator are directly sent to the input buffer area of the other party according to the output result of the load distributor.
[0028] In a specific embodiment, the load distributor can be used to compare the forward propagation delay in the first accelerator and the reverse propagation delay in the second accelerator to obtain a larger first delay; compare the backward propagation delay in the first accelerator and the forward propagation delay in the second accelerator to obtain a larger second delay; and then compare the first delay with the second delay to select the accelerator configuration scheme corresponding to the smaller delay.
[0029] In order to balance the forward propagation (FP) and back propagation (BP) calculations performed simultaneously, an embodiment of the present invention designs a scheduler that takes sparsity into consideration during the execution (EX) phase. Figure 2 The microarchitecture of this scheduler is shown, which takes 6 cycles to complete. It consists of four parts: sparse information extractor (sparse feature extractor), delay estimation unit, workload dispatcher and forward detector. The scheduler takes the layer ID (i.e. the layer number in the network) as input and generates data forward signals to the system bus and workload start signals to the MEM / WB registers as output. These signals will be sent to the IPA or ACC in the memory (MEM) stage to trigger their execution.
[0030] Sparse Feature Extractor (SFE): In SFE, a model table is used to associate the network layer ID with its sparsity ratio, sparsity pattern, and shape (i.e., the dimension of the weight matrix). The model table receives the layer IDs of the forward propagation (FP) and back propagation (BP) in the SCHE instruction (a type of computation instruction) and returns their sparsity to the forward and backward information buffers, respectively. The extracted information will be used to estimate the delay later.
[0031] Latency Estimation Unit (LEU): The runtime latency of each sparse layer is estimated by the following equation:
[0032] Estimated Layer Latency=α×β×Shape×Sparsity Ratio
[0033] Among them, α represents the sparsity pattern (partial weights, all weights, bias only, etc.) coefficient, which can be programmed before the model is deployed. Another coefficient β is used to represent the different computing performance of IPA and ACC. LEU reads the forward and backward information buffers to obtain the sparsity pattern, shape, and sparsity ratio. The shape and sparsity ratio are loaded directly into the input buffer, while the sparsity pattern controls the specific selection. After all inputs are ready, the calculated estimated delay will be sent to a delay list. The accuracy of the delay estimate does not need to be considered, the scheduler only needs to estimate which one propagates faster.
[0034] Workload Scheduler (WD): A delay list stores four delay values in order: forward propagation delay in IPA (FP-IPAL), reverse propagation delay in ACC (BP-ACCL), forward propagation delay in ACC (FP-ACCL), and reverse propagation delay in IPA (BP-IPAL). WD reads this list and compares them according to the following formula to determine the final scheduling solution (FSS):
[0035] FSS=min(max(FP-IPAL,BP-ACCL),max(FP-ACCL,BP-IPAL))
[0036] The scheduler can dispatch FP to IPA and BP to ACC, or dispatch FP to ACC and BP to IPA. The min in the formula selects the scheduling combination with smaller delay. Then, the comparison result is sent to the subsequent FD unit.
[0037] Forward Detector (FD): In order to overlap the data movement in the IPA / ACC switching, the present invention designs an FD unit. The final scheduling result is stored in a 1-bit register called the next workload indicator (next workload characterizer) (NWI), where 1'b0 indicates forward propagation (FP) in IPA and back propagation (BP) in ACC, and 1'b1 indicates BP in IPA and FP in ACC. In addition to NWI, another 1-bit register called the current workload indicator (current workload characterizer) (CWI) is introduced in this embodiment to record the status of the current acceleration unit. An exclusive OR gate (XOR gate) is applied to generate a data forwarding signal. If the forwarding signal is positive, IPA and ACC will feed their outputs to each other's input buffers to prepare for the next layer of calculation.
[0038] The embodiment of the present invention forms hybrid acceleration through the IPA (In-Pipeline Acceleration) unit and the ACC (accelerator), realizes parallel computing, and utilizes the pipeline acceleration function of the CPU; at the same time, the sparse-aware scheduler can estimate the delay based on the sparsity level, and arrange the FP and BP calculations to different acceleration units. In the scheduler of the present invention, the inter-layer delay is estimated based on the change in the sparsity ratio between different layers, and tasks are dynamically allocated to two accelerators at the layer granularity, so that better performance and higher resource utilization can be achieved.
[0039] In order to support low-cost transposition data retrieval of neural network matrices, the hybrid acceleration architecture provided in the embodiment of the present invention is also configured with a unified memory system for implementing conventional weight reading during forward propagation and transposed weight reading during backward propagation. Figure 3 for Figure 1 Unified memory mapping and read / write mode diagram of the unified memory system, combined with Figure 1 and Figure 3 In an embodiment of the present invention, the unified memory system includes: a weight memory bank (Weight Bank), an intermediate activation layer memory bank (Intermediate Layer Bank), a gradient memory bank (Gradient Bank) and a scalar memory bank (Scalar Bank). Using a bank-associated memory structure, it is possible to implement conventional forward weight retrieval during forward transmission and transposed weight retrieval during reverse transmission. In the unified memory system, four memory banks are divided: weight, intermediate activation layer, gradient and scalar memory. The highest two bits of the address are used as an index to distinguish different banks to reduce the overhead of address encoding and decoding. The address decoder sends the address to the correct bank through a multiplexer based on the highest two bits of the address, and the output multiplexer selects the correct data output based on the address of the data to be retrieved.
[0040] In the unified memory system, the present invention configures a storage bank specifically for training to adapt to various matrix access modes. The memory system unifies the conventional read mode and the transposed read mode. A storage bank includes 16 memory arrays, each of which has a read and write port. Without loss of generality, a matrix is mapped to n memory arrays for illustration, and each array stores n data. The mapping strategy is described as follows:
[0041] Mat[i][j]→Arr[(i+j)mod n][j],i,j∈[0,n-1]
[0042] For simplicity, the (j+1)th data of the (i+1)th array is called Arr[i][j], and the (i+1)th row and (j+1)th column of the matrix are called Mat[i][j]. To read the (k+1)th row or column of the matrix, each memory array reads one data according to the mapping strategy. Then the data array enters the rearrangement unit and performs k circular left shifts. As for writing, the data array first enters the rearrangement unit for circular right shifts, and then writes to the corresponding address.
[0043] Figure 3 An example of mapping a matrix in a unified memory system to a memory array with multiple data acquisition modes is shown. The blue arrows indicate the column read / write data flow, i.e., transposed read / write, while the purple arrows indicate the row read / write data flow, i.e., conventional read / write. With the help of this circularly stored memory bank, a transposed data flow for matrix read and write is implemented, which is stored in the weight bank, intermediate activation layer bank, and gradient bank, respectively. During forward propagation (FP), only the conventional data loading / storage mode is required. Since transposed loading and storage are required during back propagation (BP), both the weight bank and the intermediate activation layer bank adopt a circular storage format. By configuring a unified memory system in a hybrid acceleration architecture, an embodiment of the present invention realizes data retrieval that supports low-cost transposition of neural network matrices.
[0044] In the above embodiment, in order to effectively utilize the customized pipeline unit, the present invention also develops several customized instructions through RISC-V extension to control the newly developed hardware. As shown in Table 1, the customized instructions follow the RISC-V R type format, that is, 7bit funct7, 5bit rs2, 5bit rs1, 3bit funct3, 5bitrd, 7bit Opcode. Once Opcode==0x0B&&funct3==0x7, the customized instruction is recognized. Customized instructions include three types: data transmission, calculation and update control.
[0045] Table 1 shows the RISC-V extended instructions customized for both forward and reverse transmission.
[0046]
[0047]
[0048] SCHE generates forward signals to decide whether MVs send data to themselves or to each other. The workload start signal generated by SCHE notifies LD and ST to access the corresponding memory bank. Other instruction descriptions are also listed in Table 1.
[0049] The present invention also provides an intelligent edge chip, including the above-mentioned hybrid acceleration architecture, which can be used in edge devices such as autonomous driving devices, automated control devices, etc.
[0050] Figure 4 The schematic diagram of the timeline of the traditional architecture, the sparse training timeline without scheduling and the sparse training timeline with scheduling of the present invention is compared. Figure 4 As can be seen, the traditional TinyML architecture may involve an ML acceleration unit in a CPU or low-power processor chip. In order to achieve on-device training, the CPU or accelerator needs to execute FP (forward propagation) and BP (backward propagation) sequentially, not in parallel, such as Figure 4 As shown in (a). When dynamic sparsity is considered, the execution time of each training iteration will vary, that is, the delay is inversely proportional to the model sparsity, resulting in unpredictable training delays, which is fatal for scenarios that require stable delays. In contrast, by adding the IPA (In-Pipeline Acceleration) unit, the pipeline acceleration function of the CPU can be utilized to achieve parallel computing together with the ACC (accelerator). However, because there is no workload scheduling, for example, FP is always executed in IPA and BP is executed in ACC. Due to the imbalanced execution, idle time will occur during the execution process, resulting in limited performance improvement. As Figure 4 As shown in (b), the parallel acceleration of the i-th FP in IPA and the (i-1)-th BP in ACC always overlaps. Task dependency forces the i-th BP to not start execution until the i-th FP is completed. Therefore, scheduling between IPA and ACC is critical to achieving high utilization and performance improvement. In order to improve utilization and throughput, in the present invention, the scheduler estimates latency based on the sparsity level and arranges FP and BP calculations into different acceleration units, and the inter-layer latency is estimated based on the change in sparsity ratio between different layers.
[0051] The operation of a sparse training iteration is as follows: If the TinyML model has n layers, the scheduler is responsible for allocating the jth layer of the i-th FP and the (nj)th layer of the (i-1)-th BP to IPA and ACC. Using the equations mentioned above, the scheduler evaluates two possible allocation combinations. By comparing the latencies of the two combinations, one combination is selected for allocation, and IPA and ACC are started simultaneously to execute their workloads. If the two workloads complete execution at the same time, the resulting performance gap is zero. In most cases, the one that completes first will wait until both FP and BP are completed. Repeating the previous steps, the scheduler allocates the (j+1)th layer of the i-th FP and the (nj-1)th layer of the (i-1)-th BP to IPA and ACC.
[0052] The sparse information is updated at the end of one training iteration by the adopted dynamic sparsity scheme. After allocating FP layers and BP layers n times, the hyperparameters that record the sparse information in IPA and ACC are sent to the sparse information updater. Then, the sparse information updater feeds the latest sparse information back to the scheduler. Finally, the weights are updated and the next training iteration begins. Figure 4 (c) shows the scheduling mode timeline of the present invention and the code snippet based on the custom RISC-V instruction (see Table 1). Since tasks are dynamically allocated to two accelerators at the layer granularity, better performance and higher resource utilization can be achieved.
[0053] Figure 5 The following is a schematic diagram of the time gap comparison and hardware utilization comparison when there is / is not scheduling. Figure 5 As shown, the sparse-aware scheduler can effectively reduce the time gaps in training iterations. Figure 5 (a) shows the normalized time gaps in training. Compared with no scheduling, the sparse-aware scheduler reduces the time gaps by 3.6, 2.4, and 4.8 times on average in one training iteration for three models (MNV2, Proxy, and MCUNet). With the help of the scheduler, the ACC utilization on MCUNet (static sparse scheme) is improved from 46.30% to 98.96% ( Figure 5 (b)).
[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A sparse-aware scheduler, suitable for an intelligent edge chip with multiple accelerators, characterized in that: include: Sparse information extractor, delay estimation unit, load distributor and forward detector, The sparse information extractor is used to associate a network layer in a neural network with the sparse feature information of the network layer; The delay estimation unit is used to evaluate the delay of each network layer when the network layer runs on different accelerators according to the sparse feature information of the network layer; The load distributor is used to determine a configuration scheme of the accelerators to obtain a smaller delay based on a forward propagation delay in the first accelerator, a reverse propagation delay in the second accelerator, a backward propagation delay in the first accelerator, and a forward propagation delay in the second accelerator; The forwarding detector is used to determine whether the output of the first accelerator and the output of the second accelerator are directly sent to the input buffer area of the other party according to the output result of the load distributor.
2. The scheduler according to claim 1, characterized in that The sparse feature information includes: a sparse ratio, a sparse pattern, and the dimension of a weight matrix.
3. The scheduler according to claim 1 or 2, characterized in that: The load distributor is specifically used to compare the forward propagation delay in the first accelerator with the reverse propagation delay in the second accelerator to obtain a larger first delay; compare the backward propagation delay in the first accelerator with the forward propagation delay in the second accelerator to obtain a larger second delay; and then compare the first delay with the second delay to select an accelerator configuration scheme corresponding to the smaller delay.
4. A hybrid acceleration architecture, characterized in that: include: A plurality of accelerators and a sparse-aware scheduler as claimed in any one of claims 1 to 3, The multiple accelerators are used to support sparse training of simultaneous forward propagation and backward propagation.
5. The architecture according to claim 4, characterized in that There are two of the multiple accelerators, the first accelerator is a pipeline tightly coupled acceleration unit for executing vector multiplication and accumulation, and the second accelerator is a multiplication and addition array.
6. The architecture according to claim 5, characterized in that The pipeline tightly coupled acceleration unit is configured at the memory access pipeline stage of the central processing unit in the intelligent edge chip.
7. The architecture according to any one of claims 4 to 6, characterized in that: Also includes: Unified memory system for implementing normal read of weights during forward pass and transposed read of weights during backward pass.
8. The architecture according to claim 7, characterized in that The unified memory system includes: a weight memory bank, an intermediate activation layer memory bank and a gradient memory bank.
9. An intelligent edge chip, characterized in that: Comprising a hybrid acceleration architecture as described in any one of claims 4-8.
10. An edge device, characterized in that: Comprising an intelligent edge chip as claimed in claim 9.
Citation Information
Patent Citations
Neural network regularization bit serial calculation compression method and device
CN113902109A
Sparse neural network near storage reasoning accelerator method for realizing load balancing
CN118194939A
Weight sparsity techniques for accelerated deep learning
WO2022034542A1
