Computing single core and hybrid network extensible accelerator

Abstract

According to the computing single core and the hybrid network extensible accelerator, a selection module determines a computing mode of the computing single core according to the category of a network model layer; the data access module reads and transmits data according to the determined calculation mode to obtain access data; the sparse index module performs sparse processing on the access data according to the determined calculation mode; the logic control module designs a calculation unit connection mode and a data path according to the calculation mode; and the calculation array completes data scheduling and calculation execution according to the calculation unit connection mode and the data path. The calculation single-core design is a hardware acceleration architecture compatible with CNN and Transform calculation modes, a sparse index module of the calculation single-core design supports different sparse modes of convolution, full connection and attention at the same time, and a calculation array adapts to different calculation modes of convolution, full connection and attention; the utilization rate and the calculation speed of the hybrid network extensible accelerator designed on the basis of the calculation single core can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of neural network models, and in particular to a computing single-core and hybrid network scalable accelerator. Background Art

[0002] In recent years, the rapid development of artificial intelligence, especially deep learning technology, has enabled large-scale neural network models to be widely used in computer vision, natural language processing and other fields. However, the rapid growth of model parameters has brought huge computing and resource pressure to the acceleration of model reasoning. Since CNN has strong local feature extraction capabilities and Transformer has strong global dependency modeling capabilities, in order to simultaneously utilize the advantages of both to further improve reasoning speed, accuracy and energy efficiency, a number of CNN and Transformer hybrid network research works have emerged in recent years.

[0003] The CNN accelerator optimizes the design of the convolutional layers in the model, and uses the pipeline architecture and the parallel characteristics of the convolutional layers to improve the throughput and convolutional computing efficiency by increasing the utilization of arithmetic units and optimizing the storage system. The STICKER-IM CNN accelerator based on in-memory computing explores NN model mapping / scheduling, sparsity utilization, and efficient macro design, and significantly improves hardware utilization and performance through innovative kernel / channel sequential mapping methods and intra / inter-macro data reuse; Griffin optimizes the sparse network scenario design for the CNN accelerator to effectively utilize hardware resources. The Transformer accelerator usually designs a special storage structure for the QKV triple matrix to reduce data movement overhead. The main design content is to improve computing throughput through data reuse and optimize data flow, and design control logic for sequence data to achieve dynamic pipeline scheduling.

[0004] The above designs have achieved excellent optimization effects on their respective neural networks, but there are many deficiencies in accelerating the CNN and Transformer hybrid neural networks.

[0005] The main computing modes in the CNN model are convolution and full connection, while the main computing modes in the Transformer model are self-attention and full connection. The CNN accelerator does not support the new self-attention mechanism in the Transformer sufficiently, and its optimization for the data organization form and computing mode with strong locality is contrary to the Transformer's global data access mode and computing pipeline. The structural differences in the network also make it difficult for the original control logic design of the CNN accelerator to efficiently schedule the computing tasks in the Transformer.

[0006] The Transformer accelerator also has similar problems, and because the number of Transformer model parameters is usually larger than that of the CNN model, there will be insufficient utilization of hardware resources when the accelerator calculates CNN.

[0007] Existing hybrid accelerator research usually uses heterogeneous computing units. However, in a multi-core heterogeneous architecture, it is necessary to design data flows and control signals that adapt to multiple computing modes at the same time, which increases data migration overhead and control complexity. Secondly, when calculating a hybrid network, only the cores that calculate a specific layer are in working state, and the hardware utilization rate is low. Finally, if the connection structure of the heterogeneous computing core is in the form of a bus, as the number of cores increases, the bus will become a communication bottleneck, limiting scalability; if the connection structure is a specially designed interconnection module, it is necessary to consider data transmission in various situations, making it difficult to reduce energy consumption and hardware overhead. At present, there is an extreme lack of hardware acceleration solutions for CNN and Transformer hybrid networks, so it is imperative to study efficient hardware acceleration solutions for CNN and Transformer hybrid networks. Summary of the invention

[0008] In view of the defects in the prior art, an object of the present invention is to provide a computing single-core and hybrid network scalable accelerator.

[0009] According to one aspect of the present invention, there is provided a computing single core, comprising:

[0010] A selection module, wherein the selection module determines whether the calculation mode of the single core is convolution, full connection or attention mode according to the CNN and Transformer network model layer categories;

[0011] A data access module, wherein the data access module reads and transmits data according to a determined computing mode to obtain access data;

[0012] A sparse index module, wherein the sparse index module performs sparse processing on the access data according to a determined calculation method;

[0013] A logic control module, wherein the logic control module designs a computing unit connection mode and a data path according to a determined computing mode;

[0014] A computing array, wherein the computing array completes data scheduling and computing execution for the sparse processing results according to the computing unit connection mode and the data path.

[0015] Preferably, the data access module reads and transmits data according to the determined computing mode to obtain access data, including:

[0016] When the calculation mode is convolution and full connection, the data read and transmitted by the data access module are: input data X and weight data W;

[0017] When the calculation mode is attention mode, the data read and transmitted by the data access module are: input data X, query matrix W q , key matrix W k .

[0018] Preferably, the sparse index module independently configures the sparse mode and sparsity degree of different CNN and Transformer network model layers, wherein the sparse mode mainly includes structured sparsity and unstructured sparsity, and the sparsity degree includes the proportion of zero values ​​of input data and weight data of each model layer.

[0019] Preferably, the sparse index module performs sparse processing on the access data according to a determined calculation method, including:

[0020] For convolutional and fully connected structured sparse, the sparse index module generates a non-zero structure value index based on the structured sparse, and the calculation array performs selective skip calculation according to the non-zero structure value index;

[0021] For unstructured sparse convolution and full connection, the sparse index module obtains a sparse weight matrix based on the unstructured sparse, compresses and stores it, and when the computing array reads the sparse weight matrix in the storage format, only the activation value non-zero data is accessed for calculation; and the index lookup table is embedded in the overall architecture of the computing single core to accelerate the access process;

[0022] For the sparse key-value pairs of self-attention, the sparse index module generates a non-zero structure value index based on the sparse key-value pairs, the calculation array selectively skips calculations according to the non-zero structure value index, uses the non-zero structure value index to identify the zero item position, and controls the activation of the calculation array.

[0023] Preferably, the computing unit performs multiplication and accumulation operations on input data, and includes a selector MUX, a multiplier, a shifter, an adder and a local register connected in sequence.

[0024] Preferably, the computing array supports both a weight-fixed data multiplexing mode and an output-fixed data multiplexing mode;

[0025] The weight-fixed data reuse mode is: using the data stored in the computing unit to multiply the input data, and accumulating the product results outside the computing array;

[0026] Among them, the output fixed data multiplexing mode is: the input and weight data flow into the calculation unit, and the output is stored in the register inside each of the calculation units.

[0027] Preferably, the computing array module is adapted to three different computing modes: convolution, full connection, and attention. Specifically:

[0028] For convolution and full connection computing modes, the weight fixed mode or the output fixed mode is used;

[0029] For the attention calculation mode, the calculation formula is: QK T =(XW q )(XW k ) T =X(XW k W q T ) T =X(X(W q W k T ) T ) T =X(W q W k T X T ); The first stage uses the output fixed mode to calculate W q W k T With X T The second stage uses the weight fixed mode to read the data stored in the register and multiply it with the input data, and then accumulate it outside the calculation array to obtain QK T Calculate the result.

[0030] Preferably, the computing array includes two data transmission modes: one is pulse transmission, in which data is transmitted sequentially between computing units; the other is parallel computing, in which input data is directly broadcast to each computing unit without the need for sequential transmission between computing units.

[0031] According to a second aspect of the present invention, a hybrid network scalable accelerator is provided, comprising a two-dimensional computing core array consisting of at least two of the above-mentioned computing cores. Based on the two-dimensional computing core array, the convolution, full connection, and attention model layers are respectively mapped to different cores to achieve parallel execution between different model layers.

[0032] Preferably, computing cores at the same layer are connected via an interconnection module; each computing core has its own local cache; based on the interconnection module and the local cache, data parallelism and data scheduling between computing cores are achieved.

[0033] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0034] The computing single core and hybrid network scalable accelerator in the embodiments of the present invention are compatible with the hybrid neural network hardware acceleration architecture of CNN and Transformer computing modes, and a data access module and a control logic module are designed. The sparse index module can simultaneously support different sparse modes of convolution, full connection, and attention. The computing array can adapt to three different computing modes of convolution, full connection, and attention, which can effectively improve the utilization rate and computing speed of the hybrid network scalable accelerator designed based on the computing single core.

[0035] In the embodiment of the present invention, the computing single core and hybrid network scalable accelerator independently configure the sparse mode of different layers to cooperate with the computing unit, so that the instruction system automatically perceives the sparse information and reduces the computational complexity;

[0036] The computing single-core and hybrid network scalable accelerators in the embodiments of the present invention divide different model layers (convolutional layer, attention layer, fully connected layer) into different cores in a two-dimensional computing core array, and a larger scale of computing parallelism can be achieved through dynamic scheduling between cores.

[0037] The use of the computing single core and hybrid network scalable accelerator in the embodiments of the present invention can enable the hybrid network constructed using the complementary characteristics of CNN and Transformer to achieve higher accuracy in various tasks; it can achieve hardware acceleration of the hybrid network, improve reasoning speed and energy efficiency; further enhance the adaptability and flexibility of hardware for fast iterative computing, and provide strong support for the deployment of efficient intelligent systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 A schematic diagram of a computing single core structure in an embodiment of the present invention;

[0040] Figure 2 The data flow diagram of the PE computing unit in a single computing core in a preferred embodiment in different computing modes, (a) represents convolution and full connection computing, (b) represents the first stage of attention, and (c) represents the second stage of attention;

[0041] Figure 3 The data flow diagrams of two data transmission schemes in a PE computing unit in a computing single core in a preferred embodiment, (a) represents the systolic array implementation, and (b) represents the data broadcast form;

[0042] Figure 4Generate a row and column all-0 index map for a sparse processing Analog 1 detector in a preferred embodiment;

[0043] Figure 5 A topological diagram of a two-dimensional computing core array in a preferred embodiment. DETAILED DESCRIPTION

[0044] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0045] The purpose of the present invention is to create a neural network hardware that can simultaneously optimize the computational modes of convolutional neural networks (CNNs) and transformers. By innovatively designing and improving the four dimensions of computing units, data flows, architecture design, and topological structures, it aims to achieve efficient data reuse and full utilization of hardware resources. In addition, the hardware supports multiple sparsity processing and parallel scheduling between cores to achieve efficient acceleration of hybrid models.

[0046] Based on the above technical concept, in one embodiment of the present invention, Figure 1 As shown, a computing core is provided, which mainly includes a selection module, a data access module, a sparse index module, a logic control module and a computing array. The selection module determines whether the computing mode of the computing core is convolution, full connection or attention mode according to the CNN and Transformer network model layer categories; the data access module reads and transmits data according to the determined computing mode to obtain access data; the sparse index module performs sparse processing on the access data according to the determined computing mode; the logic control module designs the computing unit connection mode and data path according to the determined computing mode; the computing array completes data scheduling and computing execution for the sparse processing results according to the determined computing mode and computing unit connection mode and data path.

[0047] The existence of a single computing unit is insufficient: the computing power is limited, and usually only simple operations (such as a multiplication operation) can be performed. The existence of complex tasks has the following requirements: many practical applications need to process complex computing tasks, such as matrix multiplication, convolution operations, fully connected layer operations, and attention mechanisms, etc. These tasks often involve large-scale data processing and multiple computing steps. In view of the above shortcomings and needs, the above embodiment designs a computing array. The computing array connects multiple computing units together according to a specific topological structure, and can perform multiple computing operations at the same time to achieve parallel processing. This architecture allows complex tasks to be decomposed into multiple small tasks, which are completed in parallel by each computing unit, thereby significantly improving the computing speed. In addition, the computing array can design flexible communication and data transmission mechanisms to enable efficient collaboration between different units, optimize the computing process, and reduce data transmission bottlenecks.

[0048] This embodiment is based on the above-mentioned computing array and constructs a computing core that is compatible with convolutional neural network (CNN) and Transformer computing modes. The computing core is designed through the collaborative design of the data access module and the logic control unit. The computing array can simultaneously adapt to three different computing modes: convolution, full connection, and attention. The sparse index unit simultaneously supports different sparse modes of convolution, full connection, and attention, reducing the computational complexity. This embodiment makes full use of the advantages of the computing unit and maintains appropriate flexibility and scalability.

[0049] In order to realize the above technical concept, the designed computing unit will fully consider the different data calculation scales in CNN and Transformer to achieve efficient data reuse and hardware utilization. Therefore, in a preferred embodiment of the present invention, the customized computing unit performs multiplication and accumulation operations on the input data, which includes a selector MUX, a multiplier, a shifter, an adder and a local register connected in sequence. The computing array composed of customized computing units based on this structure is designed based on the weight fixed data reuse mode and the output fixed data reuse mode. Among them, the weight fixed data reuse mode refers to: using the data stored in the computing unit to multiply the output data, and accumulating the results outside the computing array. The output fixed data reuse mode means: the input and weight data flow into the computing array, and the output is stored in the register inside the computing unit.

[0050] Among them, the "fixed" in the fixed data reuse mode means that the initial weights obtained by accessing the data are fixed in the local registers of the PE (processing unit), and the input activation values ​​are transmitted or broadcast between each PE. Its "reuse" means that in this mode, the initial weights obtained by accessing the data are fixed in the local registers or caches of the PE and remain unchanged. During the entire calculation process, multiple input data will change continuously, while the weights are used repeatedly. This method avoids frequently loading weights from external memory (such as DRAM) to the computing unit, reduces the need to frequently load data from external memory, thereby saving storage bandwidth, saving time and resources for frequent loading of weights, and thus improving computing efficiency.

[0051] The "fixed" in the output fixed data multiplexing mode means that the output results of the PE (processing unit) array are fixedly stored in their respective local registers, while the input activation values ​​and weight data are transmitted or broadcast between different PEs. Its "multiplexing" refers to the cumulative multiplexing of output data, that is, in the PE array, the output results are fixed in the local registers, reused and accumulated many times. Input data and weight data are dynamically transmitted in the PE array. By reusing the data in the output registers, the PE unit can directly accumulate calculations locally, avoiding frequent writing of intermediate calculation results back to the memory or re-reading from the memory.

[0052] In output fixed mode, there are local registers in the PE unit for storing and accumulating the intermediate results of multiplication operations. This means that after each multiplication of the input activation value and the weight, the product is directly accumulated to the local registers inside the PE. These accumulator registers are fixed to store the accumulated output data until all input activation values ​​and weights are processed.

[0053] In order to further provide high data reuse and hardware utilization, in another embodiment of the present invention, in terms of the overall architecture, the data access module is used to optimize the data access mode to provide the input data required by different layers; at the same time, the control logic module is used to dynamically configure the connection and data path of the computing unit through commands to achieve the scheduling and execution of different computing modes. Through collaborative design in the three aspects of computing mode, data access and control logic, the accelerator can simultaneously adapt to the three different computing modes of convolution, full connection and attention in the data flow and computing process, and realize the efficient execution of CNN and Transformer hybrid models.

[0054] Furthermore, in a preferred embodiment, the access modes and data inputs of three different modes, namely, convolution, fully connected layer, and attention, are further described. Specifically, the calculation of convolution and fully connected is represented as XW, and the corresponding data access mode includes efficient reading and transmission of input data X and weight data W; the calculation of attention is QKT =(XW q )(XW k ) T =X(W q W k T X T ), the corresponding data access mode includes input data X, query matrix W q , key matrix W k As well as efficient reading and transmission of their transpose operations and dot product operations.

[0055] Based on the computing single core designed in the above embodiment, the data flow and computing process of the computing unit in the core will be changed accordingly. In a preferred embodiment of the present invention, the data flow of the PE unit in the core under different computing modes is provided, such as Figure 2 As shown in (a), (b), and (c), for convolution and linear layer calculations, you can use the weight stationary mode or the output stationary mode. The weight stationary mode is to use the data stored in the calculation unit to multiply the input data, and accumulate the product outside the calculation array; the output stationary mode is to multiply the input and weight, and store the result in the unit for retrieval.

[0056] For the attention calculation mode, the attention calculation formula can be improved: QK T =(XW q )(XW k ) T =X(XW k W q T ) T =X(X(W q W k T ) T ) T =X(W q W k T X T ), the first stage calculates W q W k T With X TThe second stage multiplies the result obtained in the first stage. The above calculation process can reuse the same set of calculation units. First, the calculation of the first stage can configure the PE unit to output stationary mode and save the calculation results in the registers within the unit; in the calculation of the second stage, the weight stationary mode is used to read the data stored in the register and multiply it with the input data, and then accumulate it outside the array to obtain QK T Calculation results. Thus, the computing unit realizes the adaptation of three computing modes: convolution, full connection, and attention.

[0057] Furthermore, in another embodiment, two computing unit connection modes and data paths of the logic control unit are provided. Figure 3 As shown in (a) and (b) in the figure: The first type uses a pulsating transmission method, where multiple PEs are closely connected to each other. Each PE is both a computing unit and a data transmission node, responsible for passing part of the processed data to the adjacent PE. Unidirectional or bidirectional connection: Each PE unit usually communicates with its adjacent PE through a unidirectional or bidirectional connection. Input data and weights are injected in a specific direction, and the data flows through each PE for local calculation and accumulation, and finally outputs the accumulated results. Each PE performs the next step of calculation after receiving the intermediate calculation results of the previous PE.

[0058] The second solution is parallel computing, and there is usually no directly dependent hardware connection between PEs. Each PU independently performs the assigned computing tasks and communicates with the memory system or other processors, and the results are summarized or synchronized globally. The input serial data is directly broadcast to each computing unit, and no data transmission between PE units is required, which shortens the cycle but requires horizontal accumulation, increasing the peripheral circuit overhead. When it is necessary to process multi-step calculation tasks, such as matrix multiplication, convolution operations, etc., a pulsating transmission method can be used, which has high computational efficiency, can make good use of data locality, and reduce access to external memory when processing is required. When it is necessary to process large-scale parallel processing or simple vector operations and other tasks with almost no data dependency between computing units, a parallel computing method can be used, which can complete a large number of independent tasks in a short time and improve the overall computing speed. Therefore, in the embodiment of the present invention, for convolution, full connection and attention layer calculation modes, parallel computing mode is selected.

[0059] Of course, in some other embodiments, the two computing methods may be combined to make the computing core more flexible.

[0060] There are different sparsities in the hybrid model of CNN and Transformer. In order to enable the system to further perceive sparse information and perform automatic scheduling, in a preferred embodiment, based on the above embodiments, different sparsity methods in convolution, full connection, and attention are optimized and designed to support multiple sparse situations at the same time.

[0061] Furthermore, in a preferred embodiment, a sparse index module is used to generate an index based on the input and the mask, and an index unit is used to identify the position of the zero item, and the activation of the computing unit is controlled, or the computing unit selectively skips the calculation based on the index. Specifically;

[0062] For convolutional and fully connected structured sparse, a non-zero structure value index can be generated through the sparse index module, and the computing unit selectively skips the calculation based on the index.

[0063] For convolution and fully connected unstructured sparse, a non-zero structure value index can be generated through the sparse index module; the weight matrix is ​​traversed, zero elements are removed, and a compressed sparse weight matrix is ​​obtained. The compressed sparse weight matrix is ​​used in conjunction with the sparse index generated by the sparse index module: the compressed matrix only stores non-zero values, and the position of the non-zero value in the unstructured sparse is indicated by the index. When reading the compressed weight matrix data, only non-zero values ​​are accessed.

[0064] Of course, activation values ​​(input data) may also be sparse. In sparse computing, the sparsity of activation values ​​is also utilized. Only non-zero activation values ​​will participate in subsequent calculations, and zero values ​​will be skipped. This means that not only can the weight matrix be compressed to reduce the computational burden of zero elements, but the zero elements in the activation values ​​will not be sent to the computing unit for multiplication and addition operations. Therefore, while reading the compressed weight matrix data, only the non-zero elements in the input activation values ​​will be accessed, and the part with zero activation values ​​will be skipped, thereby reducing the computational overhead. In general, the computing unit not only only accesses the non-zero elements in the weight matrix, but also skips the zero values ​​when reading the activation values, ensuring that calculations are performed only when both the weights and activation values ​​are non-zero. This method significantly improves the efficiency of sparse matrix multiplication operations.

[0065] For sparse key-value pairs in self-attention calculations, access to zero values ​​can be skipped in the same way, that is, using index units to identify the positions of zero items and control the activation of calculation units.

[0066] In order to achieve mixed support for structured and unstructured sparsity, in some preferred embodiments, the sparsity mode and sparsity degree of the convolutional layer, fully connected layer and attention layer are configured independently, where the sparsity mode mainly includes structured sparsity and unstructured sparsity, and the sparsity degree includes the proportion of zero values ​​of the input data and weight data of each layer of the neural network.

[0067] In some specific embodiments, Figure 4 As shown, Figure 1 The analog detector Analog 1detector in the processing of sparsity. When the PE calculation array adopts the output stationary data stream, you can directly skip the all-zero rows and columns in the data. Taking the all-0 rows and columns of activation or weight as an example, the corresponding bit mask is used in the process to generate the corresponding sparse index index control, and the 1 in the row and column is detected in the analog domain to identify the corresponding row / column sparse state. After that, the data is filled with 0 back to the result according to the index in the zero value inserter Zero Inserter. When the PE array adopts the weight stationary data stream, the sparse index of the weight can be directly used to read the corresponding activation value to complete the calculation. Similarly, the key-value pair sparseness in the attention calculation also uses the attention sparse index to read the corresponding key-value data for calculation.

[0068] In order to have a more comprehensive and in-depth understanding of the above-mentioned collaborative computing single core based on the customized computing unit, in this embodiment, Figure 1 As shown, a specific collaborative computing single-core example based on a customized computing unit is demonstrated.

[0069] As can be observed from the figure, it consists of three parts: the host CPU, dynamic random access memory (DRAM), and computing core circuit. The host CPU is the main control unit of the computing core, responsible for communicating with the external system, performing the main computing tasks, and controlling the operations of other modules. DRAM is used to store a large amount of data required for calculation, including weight data, activation data, and its corresponding sparse mask data (mask), etc. The sparse mask is used to identify which data needs to be involved in the calculation, which is particularly important when performing sparse calculations. As a data access module, the host CPU controls the transmission of DRAM data to the computing core on the right for calculation.

[0070] The computing core circuit is mainly composed of a PE computing array, a sparse index module, a data access module and a selection module. Among them, the PE computing array is composed of 4*4 computing units. Each PE computing unit includes a selector MUX, a multiplier, a shifter, an adder and a local register, which is responsible for performing multiplication and accumulation operations. Among them, the selector MUX is used to select between multiple input signals. According to different control signals, the selector selects the IN1 / W signal of the input computing unit or the output result of the local register for output; the multiplier is responsible for multiplying the signal output by the selector and the input signal IN2; the shifter is used to perform a shift operation on the numerical value of the multiplication result, which is used to realize multiplication or division by 2 nOperation, used to adjust the precision of the value; adder is used to perform addition operation, and the shift result is accumulated in the local register; local register is used to store intermediate calculation results within PE, such as the result of accumulator; or used as a temporary register to store input data or activation value, which can reduce the need for frequent data access to external memory, thereby improving operation speed and efficiency.

[0071] The sparse index module consists of analog 1 detector, zero inserter and index register Index Reg. Analog 1 detector and zero inserter are mainly used to eliminate zero values ​​in input data and fill zero values ​​back after PE array calculation to complete subsequent calculation. Index register Index Reg is used to store the index of data access so as to quickly find the required data when performing calculation.

[0072] The data access module consists of two buffers and the driver of the PE computing array. The two buffers are used to store the input data and output data of the computing process respectively; the driver drives and controls the transmission of input data and weight data to ensure that the data can reach the computing unit correctly and efficiently.

[0073] The selected module is the controller, which is mainly responsible for managing and coordinating the work of each module in the computing core, confirming the computing mode of the computing core, ensuring the correct transmission and processing of data, and using power gating technology to shut down idle circuits that are not involved in the calculation to save power consumption.

[0074] In addition, the computing core is equipped with a configuration register (Config Register): used to store system configuration parameters, which can be dynamically adjusted to meet different computing needs. And there is also a single instruction multiple data core (SIMDCore): responsible for executing parallel computing tasks, covering GeLU, ReLU, Sigmoid, Softmax, LN, BN operator operations, as well as POW (power), DIV (division), ADD (addition), SUB (subtraction), EXP (exponentiation), MEAN (average) arithmetic operations.

[0075] based on Figure 1 The overall structure of the control flow is as follows:

[0076] The controller communicates with the external system and confirms the computing mode executed by the computing core according to the CNN and Transformer network model layer categories.

[0077] The host CPU reads weight data, activation data, and sparse mask data from dynamic random access memory (DRAM) and transmits them to the computing core circuit;

[0078] The analog 1 detector eliminates zero values ​​in the input data. The sparsely processed input data and weight data are then transferred to the PE computing array through the input buffer and the input / weight driver to complete the parallel computing.

[0079] The Single Instruction Multiple Data Core (SIMD Core) is mainly responsible for calculating the activation function in the neural network, and collaborates with other computing cores to perform computing tasks. All calculation results are stored in the output cache.

[0080] The zero inserter fills the PE array with zero values ​​after calculation for subsequent calculations.

[0081] The parallelism of single-core computing in the above process will be limited, and different computing modes cannot be used at the same time. Considering that the number of parameters and computing power requirements in the hybrid model of CNN and Transformer are both large. Based on the same inventive concept, in other embodiments of the present invention, a hybrid network scalable accelerator is provided, including a computing core array composed of at least two computing cores based on customized computing units, and based on the two-dimensional computing core array, different model layers are mapped to different cores to achieve parallel execution between model layers. In some specific embodiments, the convolutional layers are aggregated to one core, the fully connected layers are mapped to another core, and the attention layers are mapped to other cores.

[0082] Further, such as Figure 5 As shown in the figure, the communication between computing cores can be scheduled by the host through the shared memory. Each computing core has its own local buffer, which can realize data interaction between cores and data access scheduling with the larger DRAM storage space outside the array through the interconnection module.

[0083] In the above embodiment, by allocating model layers to different computing cores, using local buffers and interconnect modules, and relying on the host to manage shared memory, the above mechanisms work together to achieve a higher level of parallel processing capabilities. At runtime, the system can achieve dynamic scheduling of model layers and data scheduling between computing cores.

[0084] Based on the hybrid network scalable accelerator in the above embodiment, the hybrid network constructed by utilizing the complementary characteristics of CNN and Transformer can achieve higher accuracy in various tasks; through computing arrays, sparse index units and multi-core acceleration topology, hardware acceleration of the hybrid network model is achieved, which can further improve the reasoning speed and energy efficiency and enhance the adaptability and flexibility of the hardware to fast iterative algorithms, providing strong support for the deployment of efficient intelligent systems.

[0085] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A computing single core, characterized in that: include: A selection module, wherein the selection module determines whether the calculation mode of the single core is convolution, full connection or attention mode according to the CNN and Transformer network model layer categories; A data access module, wherein the data access module reads and transmits data according to a determined computing mode to obtain access data; A sparse index module, wherein the sparse index module performs sparse processing on the access data according to a determined calculation mode; A logic control module, wherein the logic control module designs a computing unit connection mode and a data path according to a determined computing mode; A computing array, wherein the computing array completes data scheduling and computing execution for the sparse processing result according to the determined computing mode and the computing unit connection mode and data path.

2. A computing single core according to claim 1, characterized in that: The data access module reads and transmits data according to the determined computing mode to obtain access data, including: When the calculation mode is convolution and full connection, the data read and transmitted by the data access module are: input data X and weight data W; When the calculation mode is attention mode, the data read and transmitted by the data access module are: input data X, query matrix W q , key matrix W k .

3. A computing single core according to claim 1, characterized in that: The sparse index module independently configures the sparse mode and sparsity degree of different CNN and Transformer network model layers, wherein the sparse mode mainly includes structured sparse and unstructured sparse, and the sparsity degree includes the proportion of zero values ​​of input data and weight data of each model layer.

4. A computing single core according to claim 3, characterized in that: The sparse index module performs sparse processing on the access data according to the determined calculation method, including: For convolutional and fully connected structured sparse, the sparse index module generates a non-zero structure value index based on the structured sparse, and the calculation array performs selective skip calculation according to the non-zero structure value index; For convolution and fully connected unstructured sparse, the sparse index module obtains a sparse weight matrix based on the unstructured sparse and compresses and stores it; when the computing array reads the sparse weight matrix in the storage format, only the activation value non-zero data is accessed for calculation; and the index table lookup obtained according to the convolution or fully connected unstructured sparse is embedded in the overall architecture of the computing single core to accelerate the access process; For the sparse key-value pairs of self-attention, the sparse index module generates a non-zero structure value index based on the sparse key-value pairs, the calculation array selectively skips calculations according to the non-zero structure value index, uses the non-zero structure value index to identify the zero item position, and controls the activation of the calculation array.

5. A computing single core according to claim 1, characterized in that: The computing unit performs multiplication and accumulation operations on input data, and includes a selector MUX, a multiplier, a shifter, an adder and a local register connected in sequence, wherein: The selector MUX is used to select between multiple input signals. According to the control signals of different calculation modes, the selector selects the IN1 / W signal of the input calculation unit or the output result of the local register for output; The multiplier is responsible for performing a multiplication operation on the signal output by the selector and the input signal IN2 to obtain a multiplication result; The shifter is used to perform a shift operation on the multiplication result value, so as to realize multiplication or division by 2 n Operation, adjust the precision of the value; The adder is used to perform addition operations and accumulate the shift results of the shifter into the local register; The local register is used to store intermediate calculation results in the calculation unit, or is used as a temporary register to store input data or activation values.

6. A computing single core according to claim 5, characterized in that: The logic control unit, wherein the computing unit connection mode and the data path include two transmission modes: One uses pulsatile transmission, where data is transmitted sequentially between computing units; The other is parallel computing, where input data is directly broadcast to each computing unit without the need for sequential transmission between computing units; Among them, the three different computing modes of convolution, full connection, and attention all use parallel computing.

7. A computing single core according to claim 5, characterized in that: The computing array supports both a weight-fixed data multiplexing mode and an output-fixed data multiplexing mode; The weight-fixed data reuse mode is as follows: the weight is fixed in the local register of the computing unit, the weight is repeatedly used to multiply the input data, and the product results are accumulated outside the computing array; Among them, the output fixed data multiplexing mode is: the input and weight data flow into the calculation unit, the output is fixedly stored in the register inside each calculation unit, and the output is accumulated and multiplexed.

8. A computing single core according to claim 7, characterized in that: The computing array module is adapted to three different computing modes: convolution, full connection, and attention. Specifically: For convolution and full connection computing modes, the weight fixed mode or the output fixed mode is used; For the attention calculation mode, the calculation formula is: QK T =(XW q )(XW k ) T =X(XW k W q T ) T =X(X(W q W k T ) T ) T =X(W q W k T X T ); The first stage uses the output fixed mode to calculate W q W k T With X T The second stage uses the weight fixed mode to read the data stored in the register and multiply it with the input data, and then accumulate it outside the calculation array to obtain QK T Calculate the result.

9. A hybrid network scalable accelerator, characterized in that: It comprises at least two two-dimensional computing core arrays composed of computing cores according to any one of claims 1 to 8. Based on the two-dimensional computing core array, the convolution, full connection, and attention model layers are respectively mapped to different cores to achieve parallel execution between different model layers.

10. The hybrid network scalable accelerator according to claim 9, characterized in that: The computing cores in the same layer are connected through interconnection modules; each computing core has its own local cache; Based on the interconnection module and the local cache, data parallelism and data scheduling between computing cores are achieved.

Images