Large-model-oriented adaptive hybrid computing architecture

Abstract

The invention discloses a large-model-oriented self-adaptive hybrid computing architecture, relates to an in-memory computing technology, and provides the scheme for solving the problem that a dynamic switching mechanism is lacked in the prior art. Comprising a peripheral circuit control unit, a word line driving unit, a bit line driving unit, a clock unit, a weight input unit, a self-adaptive gating unit, a stage sensing mode control unit, a storage macro unit and a result output and post-processing unit, the storage macro-unit is composed of a plurality of storage partitions Banks, and each storage partition comprises an 8T storage unit array, a multi-stage adder tree and an accumulator; the whole memory macro-cell is cooperatively controlled to work in an SRAM mode, a CIM mode or a hybrid computing mode through the self-adaptive gating unit and the stage sensing mode control unit. The method has the advantages that efficient large model reasoning is realized through reasonable allocation, calculation and memory access resources. A self-adaptive gating unit and a stage sensing mode control unit are introduced, and the limitation of the existing architecture in the pre-filling and decoding stages in the reasoning process is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to in-memory computing technology, and in particular to an adaptive hybrid computing architecture for large models. BACKGROUND

[0002] The core computation of large models is the self-attention mechanism (Self-Attention), which is usually divided into a pre-padding stage and a decoding stage in actual inference. The former usually processes the entire sequence of user input and needs to calculate the key vectors (Key) and value vectors (Value) between all input tokens. This stage has a large amount of computation and strong data parallelism, requiring efficient parallel computing capability. The latter generates new tokens recursively, and only needs to focus on the KV vectors between the token and the previously generated tokens when generating each token, which has less computation but requires low latency and high bandwidth for memory access.

[0003] Existing large model inference acceleration technologies mainly include general-purpose architectures based on GPUs / TPUs, memory-compute integrated architectures, and heterogeneous architectures. However, when dealing with the comprehensive challenges of long sequence and long time series tasks, it is generally difficult to balance performance and energy efficiency. General-purpose computing architectures based on GPUs / TPUs have wide model compatibility, but sacrifice the key energy efficiency ratio, especially in long sequence inference. Memory-compute integrated architectures have large-scale parallel computing capability, but their efficiency and accuracy are limited when dealing with non-parallel, iterative decoding tasks generated step by step, and their energy consumption advantage is not obvious in small-scale computing scenarios. Although heterogeneous architectures achieve collaborative processing by introducing different types of computing units, they lack a dynamic switching mechanism, resulting in insufficient runtime flexibility, and task allocation is usually fixed at the design stage.

[0004] In summary, existing technologies still have deficiencies in balancing the computation and memory access characteristics of different stages in the large model inference process, making it difficult to dynamically adapt to performance and energy efficiency optimization. SUMMARY

[0005] The present application aims to provide an adaptive hybrid computing architecture for large models to solve the problems existing in the prior art.

[0006] The adaptive hybrid computing architecture for large models in the present application comprises a peripheral circuit control unit, a word line driving unit, a bit line driving unit, a clock unit, a weight input unit, an adaptive gating unit, a stage perception mode control unit, a memory-compute macro unit, and a result output and post-processing unit. The memory-compute macro unit is composed of several memory-compute partitions, each of which contains an 8T memory-compute cell array, a multi-stage addition tree, and an accumulator. The entire memory-compute macro unit is controlled by the adaptive gating unit and the stage perception mode control unit to work in SRAM mode, CIM mode, or a hybrid computing mode combining the two.

[0007] The adaptive hybrid computing architecture for large models has the advantages of efficient large model inference by reasonably allocating computing and memory resources. The adaptive gating unit and the stage-aware mode control unit are introduced to solve the limitations of existing architectures in considering two types of inference stages, thereby realizing efficient and low-energy large model inference. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a structural schematic diagram of the adaptive hybrid computing architecture in the present application.

[0009] Figure 2 is a working process schematic diagram of the adaptive hybrid computing architecture in the present application in the pre-filling stage.

[0010] Figure 3 is a working process schematic diagram of the adaptive hybrid computing architecture in the present application in the decoding stage.

[0011] Figure 4 is a structural schematic diagram of the 8T storage and computing unit in the present application.

[0012] Figure 5 is a flow schematic diagram of the adaptive gating unit and the stage-aware mode control unit working together in the present application.

[0013] Figure 6 is a computing process schematic diagram of the adaptive hybrid computing architecture in the present application.

[0014] REFERENCE NUMERALS: NMOS0-First NMOS tube, NMOS1-Second NMOS tube, NMOS2-Third NMOS tube, NMOS3-Fourth NMOS tube, NMOS4-Fifth NMOS tube, NMOS5-Sixth NMOS tube; PMOS0-First PMOS tube, PMOS1-Second PMOS tube. DETAILED DESCRIPTION

[0015] The adaptive hybrid computing architecture for large models in the present application integrates two computing paradigms of computing-intensive and storage-intensive. By dynamically scheduling and allocating computing resources and memory resources during the inference process, efficient processing and energy efficiency optimization of large model inference tasks are realized.

[0016] As Figure 1As shown, the overall architecture includes: peripheral circuit control unit, word line driving unit, bit line driving unit, clock unit, weight input unit, adaptive gating unit, stage-aware mode control unit, memory-compute macro unit and result output and post-processing unit. The memory-compute macro unit is composed of several memory-compute banks, each of which contains an 8T memory-compute cell array, a multi-stage adder tree and an accumulator. The adaptive gating unit and the stage-aware mode control unit cooperatively control the entire memory-compute macro unit to work in SRAM mode, CIM mode or a hybrid computing mode combining the two.

[0017] In the SRAM mode, read and write operations are consistent with conventional 6T-SRAM. The peripheral circuit of the SRAM array includes a precharge circuit and a sense amplifier, etc., for realizing normal read and write access.

[0018] In the CIM mode, the activation value is activated by the weight input unit, and the activation value is sent to the 8T memory-compute cell array of each memory-compute bank from the input port in a bit-serial manner to perform multiplication and addition calculation. Each 8T memory-compute cell is used to store the weight, and a group of N cells is used to represent a multi-bit weight. Each group of cells completes a bit-level multiplication operation of 1b x Nb to generate a partial product. All partial products are accumulated layer by layer through the multi-stage adder tree to form an intermediate result of the accumulated sum, and then sent to the accumulator. Finally, the accumulator calculates the accumulated result through shifting and addition operation. In the next clock cycle, new activation value bits are sent in for calculation and accumulation again until all the input activation values are processed.

[0019] The adaptive gating unit is used to adjust the on-off state of the memory-compute bank in the memory-compute macro unit in the architecture in different computing stages. According to the current workload, computing mode and clock cycle, idle modules are turned off to reduce power consumption, such as turning off unused memory-compute banks.

[0020] The stage-aware mode control unit dynamically monitors the computing load and the current inference stage, and switches the computing mode according to the load needs.

[0021] The peripheral circuit control unit is responsible for the timing and signals of the entire architecture. The word line driving unit is used to control the word line signals of several memory-computing partitions. According to the address and gating signal sent by the peripheral circuit control unit, the target row word line is driven, so that the corresponding 8T memory-computing unit is selected. The bit line driving unit controls the bit line signals of several memory-computing partitions. In the write operation, the bit line drives the data to be written into the target 8T memory unit; in the read operation, the sense amplifier detects the bit line voltage difference to output the data, and in the CIM mode, the bit line current or voltage change is used to represent the partial product information. The clock unit is responsible for providing the system working clock and the synchronization signal of each stage. The weight input unit is responsible for writing external weight data into each 8T memory-computing unit array, supporting multi-bit weight format, and configuring multiple memory-computing partitions according to the bit width of weight quantization, such as 2b, 4b, 8b, etc. The result output and post-processing unit is used to output the multiplication and addition result, and complete the self-attention calculation related operation.

[0022] Figure 2 The workflow of the pre-filling stage with a specific computing mode, such as the Attention mechanism in the Transformer model, in the adaptive hybrid computing architecture is demonstrated. This stage is usually at the beginning of generating a sequence, and the core of the pre-filling stage is to quickly process a large number of parallel input information, which belongs to a computationally intensive task. In the architecture of the present application, the computationally intensive task can be offloaded to the memory-computing macro unit for parallel execution in the CIM mode, greatly reducing the data transfer time, thereby improving the efficiency of the pre-filling stage. The main process is as follows: S01. Input sequence: the system receives input sequence data, including original text, Embedding vector, etc.

[0023] S02. Calculation of self-attention mechanism: the input sequence is X = [x1, x2,..., xn], and each xi (i = 1, 2,..., n) is a word vector. In order to calculate self-attention, three different matrices are needed: query matrix Q, key matrix K and value matrix V. Q = XWq, K = XWk, V = XWv, where Wq, Wk and Wv are learnable weight matrices. After generating Q, K and V, the attention score is calculated by the scaling dot product operation. The attention score measures the correlation between each word and other words.

[0024] S03. Pre-filling stage execution: computationally intensive, 8T memory-computing unit array stores and calculates operations related to the Attention mechanism. The CIM mode of the 8T memory-computing unit can efficiently perform matrix multiplication and accumulation operations, storing and processing weight matrices in place, and performing efficient multiplication and accumulation operations with input activation values.

[0025] S04. Multi-head attention layer output: The output matrix fuses the context information and gets the relevance of different information in the context. It allows the model to run the self-attention mechanism in multiple different subspaces simultaneously. Each attention head has its own weight matrix, and each head can focus on different features. The output matrices of all heads are spliced, and an additional linear transformation Wo is used to get the final multi-head attention output.

[0026] S05. Feed-forward network layer performs nonlinear transformation: Each sublayer is followed by a residual connection and layer normalization, and finally, the model outputs a sequence.

[0027] Figure 3 The workflow of the decoding phase for sequence generation tasks in the adaptive hybrid computing architecture is shown. This phase is usually autoregressive, generating tokens of the output sequence one by one. The key-value cache KV Cache is repeatedly accessed, which is a memory-intensive task. In the architecture of the present application, memory-intensive tasks can be offloaded to the SRAM mode of the computing macrocell, which can improve decoding efficiency and reduce overall power consumption. The main process is as follows: S11. New generated Token input and sequence expansion: output Token one by one using KV Cache, and update KV Cache with the KV pair calculated by the newly generated Token.

[0028] S12. Key-value cache update: Q = xiWq, K = XWk, V = XWv, where Wq, Wk and Wv are learnable weight matrices. After generating Q, K and V, the attention score is calculated by the scaling dot product operation. The attention score measures the relevance between each word and other words.

[0029] S13. Decoding phase execution: it is memory-intensive, and the Key and Value vectors of the previous text are needed every time a new Token is calculated. Since Tokens are generated one by one, it is not necessary to calculate the vectors of all Tokens every time, but the generated Key and Value can be reused, which can reduce a lot of computational overhead.

[0030] S14. Multi-head attention layer output: The output matrix fuses the context information and gets the relevance of different information in the context, allowing the model to run the self-attention mechanism in multiple different subspaces simultaneously. Each attention head has its own weight matrix, and each head can focus on different features. The output matrices of all heads are spliced, and an additional linear transformation Wo is used to get the final multi-head attention output.

[0031] S15. The feedforward network layer performs a nonlinear transformation: each sub-layer is followed by a residual connection and layer normalization. Finally, if the sequence termination condition is not reached, repeat the process with the newly generated Token as input; if the termination condition is reached, exit the loop and the model outputs the sequence.

[0032] The structure of the 8T storage and calculation unit in the application is as shown in the figure, which is composed of two PMOS tubes and six NMOS tubes. Figure 4 The source of the first PMOS tube PMOS0 is connected to VDD, the drain of the first PMOS tube PMOS0 is connected to the storage node Q, and the gate of the first PMOS tube PMOS0 is connected to the storage node QB. The source of the second PMOS tube PMOS1 is connected to VDD, the drain of the second PMOS tube PMOS1 is connected to the storage node QB, and the gate of the second PMOS tube PMOS1 is connected to the storage node Q. The source of the first NMOS tube NMOS0 is connected to ground, the drain of the first NMOS tube NMOS0 is connected to the storage node Q, and the gate of the first NMOS tube NMOS0 is connected to the storage node QB. The source of the second NMOS tube NMOS1 is connected to ground, the drain of the second NMOS tube NMOS1 is connected to the storage node QB, and the gate of the second NMOS tube NMOS1 is connected to the storage node Q. The source of the third NMOS tube NMOS2 is connected to the storage node Q, the drain of the third NMOS tube NMOS2 is connected to the bit line BL, and the gate of the third NMOS tube NMOS2 is connected to the word line WL. The source of the fourth NMOS tube NMOS3 is connected to the storage node QB, the drain of the fourth NMOS tube NMOS3 is connected to the bit line BLB, and the gate of the fourth NMOS tube NMOS3 is connected to the word line WL. The source of the fifth NMOS tube NMOS4 is connected to the drain of the sixth NMOS tube NMOS5 as a product output end, the drain of the fifth NMOS tube NMOS4 is connected to an activation input end, and the gate of the fifth NMOS tube NMOS4 is connected to the storage node Q. The source of the sixth NMOS tube NMOS5 is connected to ground, and the gate of the sixth NMOS tube NMOS5 is connected to the storage node QB.

[0033] ​The 8T memory-computing unit in the application combines static random access memory (SRAM) and compute-in-memory (CIM) two computing modes, and has the advantage of no read-write interference. In the SRAM mode, for the read operation, before the WL is selected, the BL and BLB are pre-charged to VDD. According to the data stored in the Q / QB node, after the WL is selected, the corresponding bit line is discharged along the storage node through the third NMOS tube NMOS2 / fourth NMOS tube NMOS3, resulting in a voltage drop. The sense amplifier captures the voltage change and outputs the result. For the write operation, the data to be written is loaded to the bit line BL and the bit line BLB, and once the WL is selected, the bit line is conditionally discharged / charged to the Q / QB node through the third NMOS tube NMOS2 / fourth NMOS tube NMOS3, covering the original stored data, and the write is completed. In the CIM mode, the weight data stored in Q / QB is directly loaded to BL / BLB, instead of traditional read-write. The input signal is activated and applied to the gate of the fifth NMOS tube NMOS4 and the sixth NMOS tube NMOS5, realizing the multiplication operation of the weight and the input, and the matrix multiplication calculation is completed through the transmission transistor logic AND gate. During the calculation process, the WL remains closed, and the third NMOS tube NMOS2 and the fourth NMOS tube NMOS3 are completely closed during the matrix multiplication, and there is no read-write interference problem. The transmission transistor logic AND gate is composed of the fifth NMOS tube NMOS4 and the sixth NMOS tube NMOS5.

[0034] For any 8T memory-computing unit, the logical true value results when performing the multiplication function under different input conditions are as shown in the following table:

[0035] It is illustrated that when the stored weight Weight and the input activation value Activation take different combinations, the product output result is weight x activation. When the weight is 0, no matter how the activation value is, the product output is always 0; when the weight is 1, the product output is equal to the activation value.

[0036] The working process of the adaptive gating unit and the stage perception mode control unit is as shown in Figure 5 , which realizes Figure 1The core component of the dynamic scheduling of the adaptive hybrid computing architecture makes the architecture have the dynamic allocation capability of computing and memory resources. The input layer is responsible for receiving system state monitoring signals such as power consumption, delay, load, storage and computing unit utilization, task stage and the like, and running feedback signals from the previous layer. The decision layer is the core of the control logic, and integrates an adaptive gating unit and a stage-aware mode control unit. The adaptive gating unit dynamically opens or closes the storage and computing partitions in the storage and computing macro unit according to the system load and clock state as needed; when it is detected that part of the storage and computing unit is in an idle or low load state, the gating logic can close the corresponding partition through clock gating or power gating mechanism, thereby realizing fine power consumption management. The stage-aware mode control unit dynamically switches the computing mode according to the inference stage and computing load characteristics, including the computing-intensive, memory-intensive or hybrid mode, to realize the balanced optimization of performance and energy efficiency. The adaptive gating unit is responsible for fine-grained power consumption control of hardware resources, and the stage-aware mode control unit is responsible for macro scheduling of the computing mode; the two work together to generate control decisions by comprehensively evaluating the current system state, task stage and hardware resource availability, and determine the optimal fine-grained configuration mode of each storage and computing partition in the storage and computing macro unit, so as to realize the efficient cooperation of the computing-intensive, memory-intensive and hybrid computing modes. The output layer generates specific control signals according to the decision results, which are used to dynamically adjust the computing mode of the next layer.

[0037] The computing flow of the adaptive hybrid computing architecture in the application is shown in Figure 6 The architecture can dynamically adjust the allocation strategy of computing and memory resources according to the inference stage of the large model, thereby improving the overall computing efficiency and system performance. The specific process is as follows: first, start computing, input the sequence after preprocessing into the hybrid computing architecture. Then, dynamically adjust the computing strategy according to the inference stage Prefill or Decode. When in the Prefill stage, the system enters the computing-intensive mode, and the scheduling module configures more storage and computing units in the storage and computing partition to the CIM working mode to support large-scale matrix operation; when in the Decode stage, the system enters the memory-intensive mode, and the scheduling module configures more storage and computing units in the storage and computing partition to the CIM working mode to improve the energy efficiency and data access efficiency. Finally, the system outputs and processes the computing result, and ends the computing.

[0038] For those skilled in the art, other various corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all these changes and deformations should belong to the protection scope of the claims of the application.

Claims

1. An adaptive hybrid computing architecture for large models, characterized in that, include: The system includes a peripheral circuit control unit, a word line driving unit, a bit line driving unit, a clock unit, a weight input unit, an adaptive gating unit, a stage-aware mode control unit, a stored-component macro unit, and a result output and post-processing unit. The stored-component macro unit consists of several stored-component partitions, each containing an 8T stored-component array, a multi-level adder tree, and an accumulator. The adaptive gating unit and the stage-aware mode control unit work together to control the entire stored-component macro unit to operate in SRAM mode, CIM mode, or a hybrid computing mode combining both.

2. The adaptive hybrid computing architecture for large models according to claim 1, characterized in that, In CIM mode, the activation value is input by the weight input unit. The activation value is sent from the input port to the 8T storage unit array of each storage partition in a bit-serial manner to perform multiplication and addition calculations. Each 8T storage unit is used to store the weight. N units are grouped together to represent multi-bit weights. Each group of units performs a 1b×Nb bit-level multiplication operation to generate partial products. All partial products are accumulated layer by layer through a multi-level addition tree to form intermediate results, which are then sent to the accumulator for accumulation. In the next clock cycle, the new activation value bits are sent in, and the calculation and accumulation are performed again until all activation values ​​of one input are processed.

3. The adaptive hybrid computing architecture for large models according to claim 2, characterized in that, The adaptive gating unit is used to adjust the on / off state of the storage partition Bank of the storage macro unit in the architecture at different computing stages, and to shut down idle modules to reduce power consumption based on the current workload, computing mode and clock cycle.

4. The adaptive hybrid computing architecture for large models according to claim 3, characterized in that, The stage perception mode control unit dynamically monitors the computing load and the current inference stage, and switches the computing mode according to the load requirements.

5. The adaptive hybrid computing architecture for large models according to claim 4, characterized in that, The peripheral circuit control unit is responsible for the timing and signals of the entire architecture; the word line driving unit controls the word line signals of several memory-based computing partitions, and drives the target row word line according to the address and strobe signals issued by the peripheral circuit control unit, so that the corresponding 8T memory-based computing unit is selected; the bit line driving unit controls the bit line signals of several memory-based computing partitions, and drives the bit line to write data to the target 8T memory unit during a write operation; during a read operation, the sensing amplifier detects the bit line voltage difference and outputs data. In CIM mode, the bit line current or voltage change is used to represent partial product information; the clock unit is responsible for providing the system operating clock and synchronization signals for each stage; the weight input unit is responsible for writing external weight data into each 8T memory-based computing unit array; the result output and post-processing unit is used to output the multiplication and addition results and complete the self-attention calculation related operations.

6. The adaptive hybrid computing architecture for large models according to claim 5, characterized in that, The weight input unit supports multi-bit weight format and can be configured with multiple storage partitions according to the bit width of weight quantization.

7. The adaptive hybrid computing architecture for large models according to claim 6, characterized in that, The 8T memory cell consists of two PMOS transistors and six NMOS transistors: The source of the first PMOS transistor (PMOS0) is connected to VDD, the drain of the first PMOS transistor (PMOS0) is connected to the memory node Q, and the gate of the first PMOS transistor (PMOS0) is connected to the memory node QB. The source of the second PMOS transistor (PMOS1) is connected to VDD, the drain of the second PMOS transistor (PMOS1) is connected to the memory node QB, and the gate of the second PMOS transistor (PMOS1) is connected to the memory node Q. The source of the first NMOS transistor (NMOS0) is grounded, the drain of the first NMOS transistor (NMOS0) is connected to the memory node Q, and the gate of the first NMOS transistor (NMOS0) is connected to the memory node QB. The source of the second NMOS transistor (NMOS1) is grounded, the drain of the second NMOS transistor (NMOS1) is connected to the memory node QB, and the gate of the second NMOS transistor (NMOS1) is connected to the memory node Q. The source of the third NMOS transistor (NMOS2) is connected to the memory node Q, the drain of the third NMOS transistor (NMOS2) is connected to the bit line BL, and the gate of the third NMOS transistor (NMOS2) is connected to the word line WL. The source of the fourth NMOS transistor (NMOS3) is connected to the memory node QB, the drain of the fourth NMOS transistor (NMOS3) is connected to the bit line BLB, and the gate of the fourth NMOS transistor (NMOS3) is connected to the word line WL. The source of the fifth NMOS transistor (NMOS4) is connected to the drain of the sixth NMOS transistor (NMOS5) and serves as the product output terminal. The drain of the fifth NMOS transistor (NMOS4) serves as the activation value input terminal. The gate of the fifth NMOS transistor (NMOS4) is connected to the storage node Q. The source of the sixth NMOS transistor (NMOS5) is grounded, and the gate of the sixth NMOS transistor (NMOS5) is connected to the memory node QB.

8. The adaptive hybrid computing architecture for large models according to claim 7, characterized in that, In CIM mode, the computation process word line WL remains off, causing the third NMOS transistor (NMOS2) and the fourth NMOS transistor (NMOS3) to be completely turned off during matrix multiplication.

Images