ANN accelerator system based on multi-core storage and calculation integration and control method

Abstract

The invention discloses an ANN accelerator system based on multi-core storage and calculation integration and a control method, and the system comprises an on-chip network module which serves as an interconnection channel between storage and calculation integration cores and is used for combining calculation results of the multiple storage and calculation integration cores; the synchronous main controller is used for sending an instruction to the decoder according to the process that the network-on-chip module combines the calculation results of the plurality of storage and calculation integrated cores; the decoder is used for receiving an instruction of the synchronous main controller and converting the instruction into a configuration parameter; the configuration register is used for storing configuration parameters and transmitting the configuration parameters to the asynchronous control module, the on-chip routing module and the storage and calculation integrated core; and the asynchronous control module is used for receiving the configuration parameters transmitted by the configuration register and performing time sequence control on the process of merging the calculation results by the network-on-chip module according to the configuration parameters. According to the method, storage and calculation integrated data flow control with reconfigurable parallelism is optimized, and the efficiency of neural network reasoning is greatly improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to an ANN accelerator system and control method based on multi-core storage and computing integration. Background Art

[0002] ANN (Artificial Neural Network) plays a vital role in various intelligent applications, especially in the field of deep learning. However, the complexity of ANN models and the sharp increase in computational complexity have put forward higher computing power and efficiency requirements for hardware. IMC (In-Memory Computing) can integrate storage and computing, greatly reducing the delay and energy consumption caused by data transmission, and has gradually become a research hotspot. At the same time, in order to cope with complex ANN computing tasks, multi-core processor architectures that can improve overall computing efficiency through parallel computing have become mainstream. In this context, a multi-core storage-computing integrated ANN accelerator architecture with synchronous and asynchronous mixed timing has been proposed, aiming to combine the high efficiency of synchronous circuits and the low power consumption advantages of asynchronous circuits to achieve high-performance and energy-friendly ANN computing acceleration.

[0003] Asynchronous memory-computation integrated circuits have shown strong competitive advantages in timing utilization efficiency and can significantly improve the throughput performance of ACIM (Analog-Compute-In-Memory). In addition, the ACIM multi-core NoC (Network-on-Chip) architecture obtained by the charge coupling method can achieve high reconfigurability of the computational parallelism of multiple ACIM cores and has the possibility of constructing higher computational parallelism. These characteristics rely on the deployment of asynchronous control schemes, but the current mainstream ANN computing acceleration architecture is based on synchronous timing design methods, which requires the design of a new ANN accelerator architecture that is compatible with asynchronous MAC engines and synchronous digital accelerator architectures and can be well embedded in the mainstream von Neumann system.

[0004] At present, the existing solutions mainly focus on two aspects:

[0005] 1. Synchronous multi-core accelerator: Synchronous multi-core architecture has been widely used in ANN accelerators. These architectures usually use global clock control to ensure synchronous operation between cores, thereby achieving efficient parallel computing. However, due to the existence of the global clock, the distribution of the clock tree and its power consumption become bottlenecks that limit system scalability and energy efficiency. In addition, the synchronous circuit is sensitive to process deviations, making power consumption and stability issues difficult to solve;

[0006] 2. Asynchronous storage and computing architecture: By eliminating the global clock, each module can independently determine the computing time, thereby reducing dynamic power consumption and the complexity brought by the clock tree. This architecture has obvious advantages in energy consumption, but due to the lack of a global clock, it is easy to introduce problems such as timing competition and deadlock. Especially in a multi-core architecture, the communication and synchronization coordination between cores become more complicated. Summary of the invention

[0007] In order to solve the deficiencies in the prior art, the present application proposes an ANN accelerator system and control method based on multi-core storage and computing.

[0008] First, an ANN accelerator system based on multi-core storage and computing is proposed, including:

[0009] The on-chip network module serves as an interconnection channel between the storage-computing integrated cores and is used to merge the calculation results of multiple storage-computing integrated cores;

[0010] A synchronous main controller, configured to issue instructions to a decoder for a process in which the on-chip network module merges calculation results of a plurality of integrated storage and computing cores;

[0011] A decoder, for receiving instructions from the synchronization master controller and converting the instructions into configuration parameters;

[0012] Configuration registers, used to store configuration parameters and pass them to the asynchronous control module, on-chip routing module and storage-computing integrated core;

[0013] The asynchronous control module is used to receive configuration parameters from the configuration register and perform timing control on the process of merging the calculation results of multiple storage-computation-in-one cores of the on-chip network module according to the configuration parameters.

[0014] Optionally, the on-chip network module includes:

[0015] Asynchronous input data reordering on-chip network, used to transfer data between memory and each storage-computing integrated core under the control of configuration parameters;

[0016] The asynchronous output data reordering on-chip network is used to reorder and transmit the data results generated by each storage and computing core under the control of configuration parameters.

[0017] Optionally, the asynchronous output data rearrangement network on chip includes:

[0018] An asynchronous parallelism compensation accumulator, used to receive and perform accumulation operations on the calculation results of the storage-computation integrated core;

[0019] The asynchronous convolution multiplexing accumulator is used to perform an accumulation operation on the accumulation result of the asynchronous parallelism compensation accumulator to realize convolution calculation.

[0020] Optionally, the asynchronous output data rearrangement network-on-chip further includes: a first-in-first-out queue for temporarily storing result data.

[0021] Optionally, the asynchronous control module includes:

[0022] The asynchronous slave controller receives configuration parameters from the configuration register, manages the operation of the asynchronous state machine, and ensures that the on-chip network module operates according to the configuration parameters;

[0023] An asynchronous state machine is used to operate on data according to the instructions and configuration parameters of the asynchronous slave controller.

[0024] Optionally, the asynchronous state machine includes:

[0025] The convolution state machine is used to determine whether the working time of the asynchronous convolution multiplexing accumulator meets the weight update delay of the ping-pong weight;

[0026] An input parallelism state machine is used to determine whether the parallelism compensation of the input asynchronous parallelism compensation accumulator satisfies the input channel of the feature graph;

[0027] The convolution time-division multiplexing state machine is used to determine whether the number of convolutions performed in the asynchronous convolution multiplexing accumulator meets the size of the convolution kernel;

[0028] The convolution pixel state machine is used to determine whether the number of pixels of the output feature map output by the convolution time division multiplexing state machine meets the resolution;

[0029] The weight update state machine is used to determine whether the weight update of the ping-pong weights has been completed.

[0030] In the second aspect, a multi-core storage and computing integrated ANN accelerator control method is proposed, which is implemented by the multi-core storage and computing integrated ANN accelerator system described in the first aspect, and includes the following steps:

[0031] The on-chip network module merges the computation results of multiple storage-computation-in-one cores;

[0032] The synchronous main controller issues instructions to the decoder for the process of the on-chip network module merging the calculation results of multiple storage-computation-in-one cores;

[0033] The decoder receives instructions from the synchronous main controller and converts the instructions into configuration parameters;

[0034] The configuration register passes the configuration parameters to the asynchronous control module;

[0035] The asynchronous control module receives the configuration parameters from the configuration register and performs timing control on the process of the on-chip network module merging the calculation results of multiple storage-computation-in-one cores.

[0036] Optionally, the on-chip network module merges the calculation results of multiple storage-computation-in-one cores through an asynchronous output data reordering on-chip network of the on-chip network module, wherein the asynchronous output data reordering on-chip network includes an asynchronous parallelism compensation accumulator and an asynchronous convolution multiplexing accumulator, and the steps include:

[0037] The data domain of the calculation results of the multiple storage-computation-in-one cores is input into the four-port double-end ping-pong cache of the asynchronous parallelism compensation accumulator, and the information domain of the calculation results of the multiple storage-computation-in-one cores is input into the counter of the asynchronous parallelism compensation accumulator;

[0038] The information domain and the data domain in the counter are accumulated with the last accumulation result data taken out from the four-port double-end ping-pong buffer of the asynchronous parallelism compensation accumulator;

[0039] If the counter determines that the accumulation is completed, the accumulation result and the information field in the counter are input into the asynchronous convolution multiplexing accumulator; otherwise, the accumulation result is written into the four-port double-ended ping-pong buffer of the asynchronous parallelism compensation accumulator to participate in the next accumulation operation.

[0040] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0041] (1) This application solves the problem of computational delay loss and control complexity multiplication caused by adopting pure asynchronous routing in the current von Neumann architecture, and optimizes the data flow control of storage and computing integration with reconfigurable parallelism;

[0042] (2) High throughput: Due to the use of asynchronous paradigm, it can compress the calculation timing and improve the throughput of MAC calculation;

[0043] (3) Strong reconfigurability: Due to the use of a multi-core NoC architecture, its computing parallelism is highly reconfigurable, that is, its output channels and output channel parameters are hardware reconfigurable;

[0044] (4) Strong compatibility: While adapting to the asynchronous storage and computing integrated core, it is also compatible with the mainstream synchronous von Neumann architecture;

[0045] (5) Strong versatility: The data flow is highly reconfigurable and adaptable to general ANN computing acceleration;

[0046] (6) Strong dynamic adaptability: The introduction of load-adaptive SIMD (Single Instruction Multiple Data) modules enables the system to dynamically adjust the calculation path according to the actual load of the input data, thereby optimizing resource utilization. This dynamic adaptability improves the processing efficiency of the system in a multi-tasking environment;

[0047] (7) Flexibility and scalability: The task allocation and operation mode of the computing core can be flexibly adjusted by configuring registers and state machines; the system can expand multiple computing cores according to actual needs to adapt to computing tasks of different scales;

[0048] (8) High efficiency in energy consumption: By combining synchronous and asynchronous circuits, it is possible to significantly reduce energy consumption while maintaining high-performance computing. The synchronous part ensures the efficient operation of the entire system, while the asynchronous part reduces the power consumption of the global clock and the complexity of the clock tree;

[0049] (9) Modular design: The modular design of the system makes it easy to maintain and upgrade. Each module can be optimized independently and work in conjunction with other modules, thus enhancing the overall stability and flexibility of the system.

[0050] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application are realized and obtained by the structures specifically pointed out in the description, claims and drawings.

[0051] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings.

[0052] Advantages of additional aspects of the present application will be given in part in the following description, and in part will become apparent from the following description, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 This is a system diagram of the ANN accelerator based on multi-core storage and computing as shown in Example 1 of the present application;

[0055] Figure 2 This is an architecture diagram of the ANN accelerator shown in the specific example in Embodiment 1;

[0056] Figure 3 This is a core computing engine architecture diagram shown in the specific example in Embodiment 1;

[0057] Figure 4 It is a schematic diagram of the output data rearrangement NoC architecture shown in the specific example of Embodiment 1;

[0058] Figure 5 This is a schematic diagram of the asynchronous state machine control process shown in the specific example in Embodiment 1;

[0059] Figure 6 This is a flow chart of the control method shown in the specific example in Example 2. DETAILED DESCRIPTION

[0060] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0061] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0062] The ANN accelerator system based on multi-core storage and computing integration proposed in this application aims to combine the advantages of the two types of architectures in the background technology. By introducing partial timing control of asynchronous circuits, it improves the performance, computing versatility and hardware reconfigurability of the core MAC engine (based on a multi-core storage and computing integration architecture) while reducing power consumption and improving robustness; at the same time, it retains the global timing characteristics of the synchronous circuit to ensure efficient communication and collaborative work between the main modules of the accelerator, and solves the energy efficiency, scalability and computational complexity problems in the prior art in terms of architecture.

[0063] Combine the following Figure 1 to Figure 6 The ANN accelerator system and control method based on multi-core storage and computing provided in this application are further described.

[0064] Embodiment 1

[0065] This application proposes an ANN accelerator system based on multi-core storage and computing. Figure 1 As shown, including:

[0066] The on-chip network module serves as an interconnection channel between the storage-computing integrated cores and is used to merge the calculation results of multiple storage-computing integrated cores;

[0067] In a feasible implementation scheme, the on-chip network module includes:

[0068] Asynchronous input data reordering on-chip network, used to transfer data between memory and each storage-computing integrated core under the control of configuration parameters;

[0069] The asynchronous output data reordering on-chip network is used to reorder and transmit the data results generated by each storage and computing core under the control of configuration parameters.

[0070] Specifically in this embodiment, the asynchronous output data rearrangement network on chip includes:

[0071] An asynchronous parallelism compensation accumulator, used to receive and perform accumulation operations on the calculation results of the storage-computation integrated core;

[0072] The asynchronous convolution multiplexing accumulator is used to perform an accumulation operation on the accumulation result of the asynchronous parallelism compensation accumulator to realize convolution calculation.

[0073] In a feasible implementation manner, the asynchronous output data rearrangement network-on-chip further includes: a first-in-first-out queue for temporarily storing result data.

[0074] A synchronous main controller, configured to issue instructions to a decoder for a process in which the on-chip network module merges calculation results of a plurality of integrated storage and computing cores;

[0075] A decoder, for receiving instructions from the synchronization master controller and converting the instructions into configuration parameters;

[0076] Configuration registers, used to store configuration parameters and pass them to the asynchronous control module, on-chip routing module and storage-computing integrated core;

[0077] The asynchronous control module is used to receive configuration parameters from the configuration register and perform timing control on the process of merging the calculation results of multiple storage-computation-in-one cores of the on-chip network module according to the configuration parameters.

[0078] The specific implementation process includes:

[0079] The asynchronous slave controller receives configuration parameters from the configuration register, manages the operation of the asynchronous state machine, and ensures that the on-chip network module operates according to the configuration parameters;

[0080] An asynchronous state machine is used to operate on data according to the instructions and configuration parameters of the asynchronous slave controller.

[0081] In a feasible implementation scheme, the asynchronous state machine includes:

[0082] The convolution state machine is used to determine whether the working time of the asynchronous convolution multiplexing accumulator meets the weight update delay of the ping-pong weight;

[0083] An input parallelism state machine is used to determine whether the parallelism compensation of the input asynchronous parallelism compensation accumulator satisfies the input channel of the feature graph;

[0084] The convolution time-division multiplexing state machine is used to determine whether the number of convolutions performed in the asynchronous convolution multiplexing accumulator meets the size of the convolution kernel;

[0085] The convolution pixel state machine is used to determine whether the number of pixels of the output feature map output by the convolution time division multiplexing state machine meets the resolution;

[0086] The weight update state machine is used to determine whether the weight update of the ping-pong weights has been completed.

[0087] It can be understood that the ping-pong weight is the accumulated result data stored in the four-port dual-end ping-pong cache.

[0088] In order to more clearly illustrate the above-mentioned solution of the ANN accelerator system based on multi-core storage and computing, the following is an example of a specific chip:

[0089] like Figure 2 As shown, this example chip includes the following modules:

[0090] 1. Core acceleration engine: It is located in the center of the entire architecture and is responsible for the main computing tasks. The core acceleration engine consists of multiple reconfigurable computing cores, which communicate with each other through the network on chip (NoC) to support multi-core parallel computing. The role of NoC is to connect the computing cores to achieve fast transmission of data and instructions.

[0091] 2. Input cache group: Located on the left and right sides of the core acceleration engine, it is divided into multiple storage modules. The memory group in the input cache group provides data storage for the core acceleration engine, supports read and write operations, and ensures fast access and exchange of data during the calculation process. Through NoC, the memory group can perform efficient data transmission with each computing core of the core acceleration engine.

[0092] 3. Asynchronous input / output data reordering NoC: This is a network dedicated to processing asynchronous data, used to reorder and transmit data between different computing cores. This module can process data generated asynchronously to ensure the correctness and efficiency of data transmission between different cores.

[0093] 4. L2 Cache (Full Synchronization): The L2 cache module is used to store a large amount of intermediate data and exchange data with the core acceleration engine and memory group in a fully synchronized manner. The L2 cache is the buffer of the entire system, which reduces frequent access to the main memory and improves the speed of data processing.

[0094] 5. Asynchronous slave controller: Located at the bottom of the architecture, it manages the control signals of the asynchronous modules. It ensures correct data transmission and task scheduling between asynchronous and synchronous modules. The asynchronous slave controller cooperates with the core acceleration engine, NoC and other modules to coordinate asynchronous operations.

[0095] In the present chip design, the asynchronous module is mainly a module in the accelerator system that specifically processes asynchronous data in the embodiment of the present application, and the on-chip network module includes an asynchronous output data reordering NoC; the decoder is arranged in the asynchronous input data reordering NoC; the asynchronous slave controller is arranged in the asynchronous control module and plays the same control role as the present example chip;

[0096] 6. Synchronous master controller: responsible for global synchronization control and managing the full synchronization part of the system. It cooperates with the secondary cache, memory group and core acceleration engine to ensure the smooth execution of synchronization operations;

[0097] It is also responsible for receiving and sending instructions for asynchronous data processing and controlling the processing of asynchronous data within the accelerator system;

[0098] 7. Synchronous general operator: It is part of the architecture and is responsible for performing some general computing tasks, usually synchronous operations. It is connected to the synchronous main controller and shares data with modules such as the secondary cache and core acceleration engine.

[0099] 8. Input and output controllers: responsible for data input and output operations. The input controller transfers external data to the memory group or directly to the core acceleration engine; the output controller outputs the calculation results to external devices or stores them in the memory group.

[0100] 9. Buffer (full synchronization): used for short-term storage of input data or intermediate results. It is connected to the core acceleration engine, memory group and synchronous main controller in a fully synchronous manner to ensure smooth data flow.

[0101] There are three types of arrows: gray bidirectional arrows represent synchronous buses; black bidirectional arrows represent asynchronous buses; and black unidirectional arrows represent control signals. The control signal from the asynchronous slave controller to the input data rearrangement NoC is mainly the pixel position of the current data packet, and the control signal to the synchronous master controller is the calculation completion signal; the control signal from the synchronous master controller to the configuration register is configuration information, and the control signal to the asynchronous slave controller is the instruction signal, and it controls the data interaction between the cache and the secondary cache.

[0102] The information stored inside the input cache group is a group of data (one group corresponds to several channels) on the corresponding pixel point in the processed feature map, which does not meet the data packet format. Therefore, the encoder on the asynchronous input data rearrangement NoC will solve the x bit of the data packet based on the block number and block address to which the output data of the input swap group belongs, the mapping information of the configuration register in the input cache group, and the feature map dimension information stored in the register group. For each storage and computing integration of the two-dimensional reconfigurable multi-core storage and computing integration array, there are corresponding registers in the register group to record the configuration information of the current storage and computing core, including the corresponding output channel range and quantization configuration information. In addition, the register group also stores the overall computing parallelism reconstruction information of the multi-core architecture, which will also be provided to the asynchronous output data rearrangement NoC.

[0103] The asynchronous output data reordering NoC will process the corresponding pixel space information, channel information, etc. in the storage-computing integrated output data packet, and decide whether the data is transmitted to the FIFO or written back to the mirror output cache according to the relevant configuration of the data packet info information. The former will be transmitted back to the synchronous architecture, such as the cache or the second-level cache for temporary storage or for data sharing with other computing architectures, and the latter is used to quickly realize the acceleration of two consecutive layers of ANN calculations (at this time, the data in the mirror input cache will be directly used for the next layer of calculation). For the latter, the output data reordering NoC will pass through a decoder and resolve the Info information into the cache block number and block address in the input cache group.

[0104] Due to the limitations of on-chip cache, sometimes the weight information of the ANN network cannot be stored in the on-chip cache. Therefore, a weight memory is deployed to store part of the weight information. When the weights in the multi-core storage and computing array need to be updated, the new weights can be obtained from the weight memory in time without obtaining data from the slow external memory; and when the cache in the array works normally, the weight memory can obtain the data required for the next update from the external memory. In this way, the time cost of weight updates is reduced.

[0105] Since the multi-core storage and computing architecture can calculate multiple input channels and output channels at the same time, the BN operator must support vector operations, otherwise it will block the data flow, so it is implemented using SIMD (instructions are stored in register groups and used to reconstruct different addends and multipliers);

[0106] Among them, the control signal from the synchronous master controller to the configuration register is converted into configuration information by the instruction sent by the synchronous master controller to the decoder in the asynchronous input data rearrangement NoC, and is transmitted to the asynchronous slave controller in this example. Finally, the asynchronous state machine therein realizes the rearrangement and optimization of the calculation result data on the asynchronous output data rearrangement NoC; and the instruction signal from the synchronous master controller to the asynchronous slave controller mainly serves as a trigger signal to start the accelerator system for asynchronous data processing.

[0107] Each module communicates and collaborates with each other through interconnect signals to ensure that the system operates efficiently under the conditions of synchronous and asynchronous mixed timing. NoC plays a key role here as a communication bridge between the core acceleration engine and different modules. The asynchronous slave controller and the synchronous controller manage the operation timing of different parts respectively and ensure the correct transmission of data between asynchronous and synchronous modules. Through this modular design, the entire architecture can find a balance between high performance and low power consumption.

[0108] The module design in the core acceleration engine is as follows: Figure 3 As shown, it includes multiple computing units and NoC (network on chip) modules. The following is a detailed description of each part:

[0109] Storage-computing integrated core: The gray blocks in the figure represent the computing units in the core acceleration engine, which are the main computing modules of the entire accelerator. These computing units designed based on the asynchronous paradigm can independently perform specific computing tasks, that is, receive data on the input bus, perform calculations and output the results to the bus. The calculation of the storage-computing integrated core is based on the charge paradigm.

[0110] NoC module: The NoC module is the interconnection channel between each computing unit. It is responsible for charge coupling between computing units to merge the computing results of the multi-core storage and computing core, thereby maximizing the computing parallelism of the multi-core architecture and improving the flexibility of computing resource partitioning.

[0111] Input asynchronous bus: Located on the left side of the figure, the input asynchronous bus is responsible for delivering external input data to each storage-computing integrated core. This bus system is designed to be asynchronous, which means that the input data can be independent of the global clock, thereby reducing the delay in the data transmission process and reducing power consumption. The use of the asynchronous bus makes the entire multi-core architecture more flexible and efficient in processing input data.

[0112] Output asynchronous bus (bottom gray rectangle): located at Figure 3 At the bottom of the CPU, the output asynchronous bus is responsible for outputting the calculation results of the storage-computing core to the outside. The bus system is also designed to be asynchronous, which means that multiple storage-computing cores can use the arbitration mechanism of the asynchronous bus when competing for bus resources without the need for complex additional control circuits. In addition, the asynchronous bus can better adapt to the asynchronous storage-computing working data flow, thereby maintaining the outstanding performance of the asynchronous storage-computing core in throughput. The use of the asynchronous bus makes the entire multi-core architecture more flexible and efficient when dealing with data output competition.

[0113] The entire architecture closely connects the computing units together through NoC to achieve efficient coupling of charges and flexible reconfiguration of computing resources. The input asynchronous bus provides a flexible data input channel for these integrated computing cores, while the output asynchronous bus arbitrates the output and transmission of the results of the integrated computing cores. Through this design, the core acceleration engine can maintain a balance between high efficiency and low power consumption in high-performance computing, and is suitable for ANN hardware acceleration tasks that require extremely high computing power. At the same time, its multi-core architecture allows configuration to achieve a variety of input parallelism and output parallelism, which is suitable for ANN general computing tasks that require high hardware reconfigurability.

[0114] Figure 4 The architecture diagram of the output data rearrangement NoC is shown, which mainly includes an asynchronous parallelism compensation accumulator and an asynchronous convolution multiplexing accumulator. The detailed description is as follows:

[0115] 1. Input bus (left side): The left side of the figure is the input bus, corresponding to Figure 3 The output asynchronous buses in the . These input buses input data from the core computing engine to the asynchronous parallelism compensation accumulator. The input data is transmitted to the accumulator module in parallel through each column bus, supporting parallel processing of multiple data.

[0116] 2. Asynchronous parallelism compensation accumulator (middle left): This module is responsible for processing the parallel input data stream, and achieves a result with greater computational parallelism by adding the results of several core acceleration engines. The figure shows four accumulators, corresponding to four parallel data inputs. Each accumulator receives data from the corresponding input column and performs accumulation operations on it. Through this module, the input parallelism compensation can be performed on the calculation results of the core calculation engine to improve the functional scalability of the accelerator in the key functional indicator of the input channel.

[0117] Although the current two-dimensional reconfigurable storage and computing (i.e., storage and computing integrated core) can reconstruct internal functions to adjust the computing parallelism, compared with highly diversified algorithms, its computing input parallelism is limited, and it is necessary to use the method of accumulating the results after multiple calculations to supplement the computing input parallelism (input channel), and to supplement the output parallelism (output channel) by repeated calculations; the convolution kernel dimension SR is implemented in a time-division multiplexing manner, so it also needs accumulation, and the accumulation method is the same as the accumulation method of the input channel, and the work of the accumulator involves data access, mainly involving repeated reading of data and writing back after accumulation, but this brings about the need for weight update, which is achieved through data interaction between the weight memory and the two-dimensional reconfigurable storage and computing integrated;

[0118] 3. Asynchronous convolution multiplexing accumulator (middle right): The asynchronous convolution multiplexing accumulator is responsible for further processing the output data from the asynchronous parallelism compensation accumulator, and realizes two-dimensional convolution by adding the accumulated results after parallelism compensation. This module is also composed of four accumulators, each of which takes the result from the asynchronous parallelism compensation accumulator as input and performs a convolution operation. This module supports the resizing of the convolution kernel through the two-dimensional traversal operation of the convolution kernel, so as to more efficiently process the computational tasks in the convolutional neural network.

[0119] Due to the particularity of the two-dimensional reconfigurable storage and computing integrated circuit, the input bandwidth (Bwi) and output bandwidth (Bwo) of the multi-core architecture will change dynamically, and the constraint Bwi*BW=Const is satisfied. However, the number of resources such as accumulators and SIMDs is unchanged, so there is a distribution between the output bandwidth requirement and the working capacity of the computing module (accumulators, SIMDs, etc.). The proportional relationship between them requires a many-to-many arbitration mechanism to match this difference. Otherwise, the core computing engine will significantly reduce the utilization of the internal two-dimensional reconfigurable storage and computing multi-core array due to frequent jams in the data flow, resulting in performance degradation.

[0120] Therefore, the cache and accumulator structure is constructed through the asynchronous paradigm, thereby improving the data flow adaptability of the computing module (the asynchronous paradigm is not sensitive to timing, so it can match different bandwidth requirements without changing the circuit resource overhead).

[0121] 4. Load-adaptive SIMD (right): Immediately following the asynchronous convolution multiplexing accumulator is the load-adaptive SIMD module. This module dynamically adapts to the load of input data through LUT (lookup table) and matching register reconfigurable control logic to optimize computing efficiency. Through adaptive control, the system can flexibly adjust the data processing path under different computing loads to improve energy efficiency.

[0122] 5. FIFO or asynchronous bus (far right): After the calculation is completed, the output data can go through FIFO (first-in-first-out queue) or directly enter the asynchronous bus. FIFO is used to temporarily store the calculation results to ensure the orderly output of data, while the asynchronous bus is used to directly transmit the results to the next level processing module or external interface. This design allows the system to effectively cope with different processing speeds and output requirements when processing large-scale data.

[0123] During operation, the result of multi-core storage and computing is provided to the output data rearrangement NoC through four column buses, and the data on it is divided into two domains: data domain and information domain. The data domain is used to record the result of multi-core storage and computing, and the information domain is used to record the output channel, pixel coordinates and other information corresponding to this result. At the input end of the NoC, the data domain is passed to the a1 bus, and the information domain is passed to the a2 bus. Subsequently, the information domain data is input into the accumulator with completion criteria, and at the same time, the data in the data domain (through bus b) is used as one of the addends and is accumulated with the last accumulation result out_data[t-1] taken out from the four-port dual-end ping-pong buffer (through bus c). After accumulating the results, if the accumulator determines that the parallelism compensation accumulation is completed, the result and the information domain are passed into the asynchronous convolution multiplexing accumulator (through buses f1 and f2); otherwise, the result out_data[t] is written into the four-port dual-end ping-pong buffer (through bus e) to participate in the next accumulation operation.

[0124] In the asynchronous convolution multiplexing accumulator, the processing principle is the same, but the difference is that the judgment criterion of the accumulator is whether the traversal calculation of the convolution kernel is completed.

[0125] Subsequently, through the j bus, the result of the convolution accumulation and the information domain are packaged into a Conv_result and transmitted to the load adaptive SIMD module (with four bus parallel inputs), and the activation function (based on LUT) calculation and Batch Normalization (BN) operation are completed in this module (the activation and BN operation functions can be reconfigured through the internal register group), and the related operation bit vector calculation is used to improve the computing power. Subsequently, the SIMD calculation result and the corresponding information domain are packaged into Final_result and will be output to the FIFO or asynchronous bus according to the information in the information domain.

[0126] In summary, this architecture diagram shows the complete data flow from input data, parallel processing, accumulation, convolution operation, to load adaptive processing. Through the combination of asynchronous parallelism compensation accumulator and asynchronous convolution multiplexing accumulator, this accelerator architecture can efficiently process complex convolution operations, and implement adaptive data load management through SIMD modules, and finally output the results to the next level module or external through FIFO or asynchronous bus. This design greatly improves the efficiency of neural network reasoning and is suitable for high-performance computing tasks.

[0127] Figure 5 A schematic diagram of the asynchronous state machine control process is shown, involving the convolution state machine, convolution time-division multiplexing state machine, convolution pixel state machine, input parallelism state machine, and weight update state machine. The following is a detailed description of the process:

[0128] 1. Convolutional state machine

[0129] IDLE state: The convolution state machine is initially in an idle state, waiting for instructions to start operation.

[0130] CAL state: After receiving the instruction, the state machine enters the calculation state and performs the convolution operation.

[0131] Conditional judgment (Pixel>H*W): The convolution state machine determines whether the number of processed pixels exceeds the product of the height and width of the image (i.e. the total number of pixels). If not, the convolution calculation continues; if it exceeds, it enters the end state.

[0132] End state: When the convolution operation is completed, the convolution state machine enters the end state and completes the current task.

[0133] 2. Convolutional Time Division Multiplexing State Machine

[0134] IDLE state: The convolutional time division multiplexing state machine is initially in the idle state.

[0135] CAL state: When entering the convolution time-division multiplexing operation, the state machine enters the calculation state to process parallel convolution operations.

[0136] Conditional judgment (CNT>S*R): The state machine judges whether the current number of convolutions exceeds the product of the specified time and space dimensions. If not, the operation continues; if it exceeds, the state enters the end state.

[0137] End state: When the convolution time division multiplexing is completed, the state machine enters the end state.

[0138] 3. Input parallelism state machine

[0139] IDLE state: The input parallelism state machine is initially in the idle state.

[0140] CAL state: Enters the parallel processing state and processes the parallelism of input data.

[0141] Conditional judgment (CNT>x / a): The state machine judges whether the current input counter value exceeds the threshold x / a. If not, the processing continues; if it exceeds, it enters the end state.

[0142] End state: After the processing is completed, the state machine enters the end state.

[0143] 4. Convolutional Pixel State Machine

[0144] IDLE state: The convolution pixel state machine is initially in the idle state.

[0145] CAL state: Enters the calculation state to process the convolution operation of each pixel.

[0146] Conditional judgment (is CNT greater than the threshold and the weight update completed?): The state machine determines whether the current counter exceeds the threshold and checks whether the weight update is completed. If the condition is met, it enters the end state; otherwise, it continues the operation.

[0147] End state: After the convolution pixel operation is completed, the state machine enters the end state.

[0148] 5. Update the state machine

[0149] IDLE state: The weight update state machine is initially in the idle state.

[0150] CAL state: Enter the calculation state and update the convolution weights.

[0151] Conditional judgment (weight update completed?): The state machine determines whether the weight update has been completed. If completed, it enters the end state; otherwise, it continues to update.

[0152] End state: After the weight update is completed, the state machine enters the end state.

[0153] Figure 5 The interaction process between different state machines is described. The state machines determine the transition to the next state through conditional judgment, ensuring that the entire convolution operation and related computing tasks are completed efficiently in sequence. The figure shows that the system gradually enters the computing state from the initialized idle state, returns to the idle state or enters the end state after completing each task, so as to prepare for processing the next set of data or operations. This design realizes distributed control of complex operations and is suitable for parallel computing and efficient general convolutional neural network processing tasks.

[0154] Embodiment 2

[0155] The present application proposes a multi-core storage and computing integrated ANN accelerator control method, which is implemented by the multi-core storage and computing integrated ANN accelerator system described in Example 1, and includes the following steps:

[0156] Step S1: The on-chip network module merges the calculation results of multiple storage-computation-in-one cores;

[0157] Step S2: The synchronization main controller issues an instruction to the decoder regarding the process of the on-chip network module merging the calculation results of multiple storage-computation-in-one cores;

[0158] Step S3: The decoder receives the instruction from the synchronous main controller and converts the instruction into configuration parameters;

[0159] Step S4: The configuration register transmits the configuration parameters to the asynchronous control module;

[0160] Step S5: The asynchronous control module receives the configuration parameters transmitted from the configuration register and performs timing control on the process of the on-chip network module merging the calculation results of multiple storage-computation-in-one cores.

[0161] In a feasible implementation scheme, the on-chip network module merges the calculation results of multiple storage-computation-in-one cores through the asynchronous output data rearrangement on-chip network of the on-chip network module, wherein the asynchronous output data rearrangement on-chip network includes an asynchronous parallelism compensation accumulator and an asynchronous convolution multiplexing accumulator, and the steps include:

[0162] Step S1.1: the data domain of the calculation results of the multiple storage-computing-in-one cores is input into the four-port double-ended ping-pong cache of the asynchronous parallelism compensation accumulator, and the information domain of the calculation results of the multiple storage-computing-in-one cores is input into the counter of the asynchronous parallelism compensation accumulator;

[0163] Step S1.2: the information field and the data field in the counter are accumulated with the last accumulation result data taken out from the four-port double-ended ping-pong buffer of the asynchronous parallelism compensation accumulator;

[0164] Step S1.3: If the counter determines that the accumulation is completed, the accumulation result and the information field in the counter are input into the asynchronous convolution multiplexing accumulator; otherwise, the accumulation result is written into the four-port double-ended ping-pong cache of the asynchronous parallelism compensation accumulator to participate in the next accumulation operation.

[0165] In order to more clearly illustrate the above-mentioned ANN accelerator control method based on multi-core storage and computing integration, a specific control example is given below to illustrate. Figure 6 As shown:

[0166] The synchronous master controller is responsible for the overall control of the system, issues instructions to the next module, and coordinates the operations of each module. This controller ensures the timing control of the synchronous part of the system;

[0167] The decoder receives instructions from the synchronous master controller and converts them into specific configuration parameters;

[0168] Specifically, these parameters include image 2D size, padding, stride, input channel, output channel, convolution kernel size, operation mode, etc. These data are stored in configuration registers and are used to pass parameters to the asynchronous slave controller and asynchronous state machine to control their behavior, as detailed below:

[0169] 1. Image two-dimensional size (H, W): 20 bits, used to represent the height (H) and width (W) of the image, and can support images with a maximum resolution of 1024x1024;

[0170] 2.Pad: 2 bits, used to set the Padding parameter, 0 means no padding operation;

[0171] 3.stride(str): 1 bit, used to set the stride of convolution, 0 means the stride is 1;

[0172] 4. Input channel (InCh): 6 bits, used to specify the number of input channels. The range of input channels is 16 to 1024. Each Core supports 2 degrees of parallelism, and each write is divided into 16 degrees of parallelism, so the maximum supported input channels is 1024;

[0173] 5. Output channel (OutCh): 6 bits, used to specify the number of output channels, the range is also 16 to 1024, each Core supports 16 output channels;

[0174] 6. Convolution kernel size (Conv(S,R)): 3 bits, used to set the size of the convolution kernel, which is square by default. If set to 0, the convolution kernel size is 1x1;

[0175] 7. Operation mode (Mode): 2 bits, used to set the operation mode;

[0176] 0 means using only one input buffer and outputting to external memory; 1 means outputting to another output buffer;

[0177] The asynchronous slave controller receives the control signal from the configuration register and is responsible for the timing control of the asynchronous data processing part;

[0178] Specifically, the asynchronous slave controller mainly manages the operation of the asynchronous state machine to ensure that the asynchronous module performs correct calculation operations according to the configuration parameters. The asynchronous state machine performs specific tasks such as convolution calculation, data transmission, etc. according to the instructions and configuration parameters of the asynchronous slave controller.

[0179] Through this configuration, the system can flexibly control the behavior of the accelerator to adapt to different computing task requirements. The configuration parameters are synchronized with the main controller and the decoder to ensure that the accelerator can efficiently and accurately complete various operations during the convolutional neural network reasoning process and is compatible with the mainstream von Neumann computer architecture.

[0180] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. An ANN accelerator system based on multi-core storage and computing, characterized in that: include: The on-chip network module serves as an interconnection channel between the storage-computing integrated cores and is used to merge the calculation results of multiple storage-computing integrated cores; A synchronous main controller, configured to issue instructions to a decoder for a process in which the on-chip network module merges calculation results of a plurality of integrated storage and computing cores; A decoder, for receiving instructions from the synchronization master controller and converting the instructions into configuration parameters; Configuration registers, used to store configuration parameters and pass them to the asynchronous control module, on-chip routing module and storage-computing integrated core; The asynchronous control module is used to receive configuration parameters from the configuration register and perform timing control on the process of merging the calculation results of multiple storage-computation-in-one cores of the on-chip network module according to the configuration parameters.

2. The ANN accelerator system based on multi-core storage and computing integration according to claim 1 is characterized in that: The on-chip network module comprises: Asynchronous input data reordering on-chip network, used to transfer data between memory and each storage-computing integrated core under the control of configuration parameters; The asynchronous output data reordering on-chip network is used to reorder and transmit the data results generated by each storage and computing core under the control of configuration parameters.

3. The ANN accelerator system based on multi-core storage and computing integration according to claim 2 is characterized in that: The asynchronous output data rearrangement network on chip comprises: An asynchronous parallelism compensation accumulator, used to receive and perform accumulation operations on the calculation results of the storage-computation integrated core; The asynchronous convolution multiplexing accumulator is used to perform an accumulation operation on the accumulation result of the asynchronous parallelism compensation accumulator to realize convolution calculation.

4. The ANN accelerator system based on multi-core storage and computing integration according to claim 2, characterized in that: The asynchronous output data rearrangement network-on-chip also includes a first-in-first-out queue for temporarily storing result data.

5. The ANN accelerator system based on multi-core storage and computing integration according to claim 3 is characterized in that: The asynchronous control module comprises: The asynchronous slave controller receives configuration parameters from the configuration register, manages the operation of the asynchronous state machine, and ensures that the on-chip network module operates according to the configuration parameters; An asynchronous state machine is used to operate on data according to the instructions and configuration parameters of the asynchronous slave controller.

6. The ANN accelerator system based on multi-core storage and computing integration according to claim 5, characterized in that: The asynchronous state machine comprises: The convolution state machine is used to determine whether the working time of the asynchronous convolution multiplexing accumulator meets the weight update delay of the ping-pong weight; An input parallelism state machine is used to determine whether the parallelism compensation of the input asynchronous parallelism compensation accumulator satisfies the input channel of the feature graph; The convolution time-division multiplexing state machine is used to determine whether the number of convolutions performed in the asynchronous convolution multiplexing accumulator meets the size of the convolution kernel; The convolution pixel state machine is used to determine whether the number of pixels of the output feature map output by the convolution time division multiplexing state machine meets the resolution; The weight update state machine is used to determine whether the weight update of the ping-pong weights has been completed.

7. A control method for an ANN accelerator based on multi-core storage and computing integration, implemented by using the ANN accelerator system based on multi-core storage and computing integration according to any one of claims 1 to 6, characterized in that: The steps include: The on-chip network module merges the computation results of multiple storage-computation-in-one cores; The synchronous main controller issues instructions to the decoder for the process of the on-chip network module merging the calculation results of multiple storage-computation-in-one cores; The decoder receives instructions from the synchronous main controller and converts the instructions into configuration parameters; The configuration register passes the configuration parameters to the asynchronous control module; The asynchronous control module receives the configuration parameters from the configuration register and performs timing control on the process of the on-chip network module merging the calculation results of multiple storage-computation-in-one cores.

8. The control method of the ANN accelerator based on multi-core storage and computing integration according to claim 7, characterized in that: The on-chip network module combines the calculation results of multiple storage-computation integrated cores through the asynchronous output data rearrangement on-chip network of the on-chip network module, wherein the asynchronous output data rearrangement on-chip network includes an asynchronous parallelism compensation accumulator and an asynchronous convolution multiplexing accumulator, and the steps include: The data domain of the calculation results of the multiple storage-computation-in-one cores is input into the four-port double-end ping-pong cache of the asynchronous parallelism compensation accumulator, and the information domain of the calculation results of the multiple storage-computation-in-one cores is input into the counter of the asynchronous parallelism compensation accumulator; The information domain and the data domain in the counter are accumulated with the last accumulation result data taken out from the four-port double-end ping-pong buffer of the asynchronous parallelism compensation accumulator; If the counter determines that the accumulation is completed, the accumulation result and the information field in the counter are input into the asynchronous convolution multiplexing accumulator; otherwise, the accumulation result is written into the four-port double-ended ping-pong buffer of the asynchronous parallelism compensation accumulator to participate in the next accumulation operation.

Images