An adaptive hybrid computing architecture for large models

Abstract

The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the problem that the prior art lacks a dynamic switching mechanism. The application discloses an adaptive hybrid computing architecture for large models, relates to in-memory computing technology, and aims at the

Description

TECHNICAL FIELD

[0001] The present application relates to in-memory computing technology, and in particular to an adaptive hybrid computing architecture for large models. BACKGROUND

[0002] The core computation of large models is the self-attention mechanism (Self-Attention), which is usually divided into a pre-padding stage and a decoding stage in actual inference. The former usually processes the entire sequence of user input and needs to calculate the key vectors (Key) and value vectors (Value) between all input tokens. This stage has a large amount of computation and strong data parallelism, requiring efficient parallel computing capability. The latter generates new tokens recursively, and only needs to focus on the KV vectors between the token and the previously generated tokens when generating each token, which has less computation but requires low latency and high bandwidth for memory access.

[0003] Existing large model inference acceleration technologies mainly include general-purpose architectures based on GPUs / TPUs, memory-compute integrated architectures, and heterogeneous architectures. However, when dealing with the comprehensive challenges of long sequence and long time series tasks, it is generally difficult to balance performance and energy efficiency. General-purpose computing architectures based on GPUs / TPUs have wide model compatibility, but sacrifice the key energy efficiency ratio, especially in long sequence inference. Memory-compute integrated architectures have large-scale parallel computing capability, but their efficiency and accuracy are limited when dealing with non-parallel, iterative decoding tasks generated step by step, and their energy consumption advantage is not obvious in small-scale computing scenarios. Although heterogeneous architectures achieve collaborative processing by introducing different types of computing units, they lack a dynamic switching mechanism, resulting in insufficient runtime flexibility, and task allocation is usually fixed at the design stage.

[0004] In summary, existing technologies still have deficiencies in balancing the computation and memory access characteristics of different stages in the large model inference process, making it difficult to dynamically adapt to performance and energy efficiency optimization. SUMMARY

[0005] The present application aims to provide an adaptive hybrid computing architecture for large models to solve the problems existing in the prior art.

[0006] The adaptive hybrid computing architecture for large models in the present application comprises a peripheral circuit control unit, a word line driving unit, a bit line driving unit, a clock unit, a weight input unit, an adaptive gating unit, a stage perception mode control unit, a memory-compute macro unit, and a result output and post-processing unit. The memory-compute macro unit is composed of several memory-compute partitions, each of which contains an 8T memory-compute cell array, a multi-stage addition tree, and an accumulator. The entire memory-compute macro unit is controlled by the adaptive gating unit and the stage perception mode control unit to work in SRAM mode, CIM mode, or a hybrid computing mode combining the two.

[0007] The adaptive hybrid computing architecture for large models has the advantages of efficient large model inference by reasonably allocating computing and memory resources. The adaptive gating unit and the stage-aware mode control unit are introduced to solve the limitations of existing architectures in considering two types of inference stages, thereby realizing efficient and low-energy large model inference. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a structural schematic diagram of the adaptive hybrid computing architecture in the present application.

[0009] Figure 2 is a working process schematic diagram of the adaptive hybrid computing architecture in the present application in the pre-filling stage.

[0010] Figure 3 is a working process schematic diagram of the adaptive hybrid computing architecture in the present application in the decoding stage.

[0011] Figure 4 is a structural schematic diagram of the 8T storage and calculation unit in the present application.

[0012] Figure 5 is a flow schematic diagram of the adaptive gating unit and the stage-aware mode control unit working together in the present application.

[0013] Figure 6 is a computing process schematic diagram of the adaptive hybrid computing architecture in the present application.

[0014] REFERENCE NUMERALS:

[0015] NMOS0-First NMOS tube, NMOS1-Second NMOS tube, NMOS2-Third NMOS tube, NMOS3-Fourth NMOS tube, NMOS4-Fifth NMOS tube, NMOS5-Sixth NMOS tube;

[0016] PMOS0-First PMOS tube, PMOS1-Second PMOS tube. DETAILED DESCRIPTION

[0017] The adaptive hybrid computing architecture for large models in the present application integrates two computing paradigms of computation-intensive and storage-intensive. By dynamically scheduling and allocating computing resources and memory resources during the inference process, efficient processing and energy efficiency optimization of large model inference tasks are realized.

[0018] As Figure 1As shown, the overall architecture includes: peripheral circuit control unit, word line driving unit, bit line driving unit, clock unit, weight input unit, adaptive gating unit, stage-aware mode control unit, memory-compute macro unit and result output and post-processing unit. The memory-compute macro unit is composed of several memory-compute banks, each of which contains an 8T memory-compute cell array, a multi-stage adder tree and an accumulator. The adaptive gating unit and the stage-aware mode control unit cooperatively control the entire memory-compute macro unit to work in SRAM mode, CIM mode or a hybrid computing mode combining the two.

[0019] In the SRAM mode, read and write operations are consistent with conventional 6T-SRAM. The peripheral circuit of the SRAM array includes a precharge circuit and a sense amplifier, etc., for realizing normal read and write access.

[0020] In the CIM mode, the activation value is activated by the weight input unit, and the activation value is sent to the 8T memory-compute cell array of each memory-compute bank from the input port in a bit-serial manner to perform multiply-add computation. Each 8T memory-compute cell is used to store the weight, and a group of N cells is used to represent a multi-bit weight. Each group of cells completes a bit-level multiplication operation of 1b x Nb to generate a partial product. All partial products are accumulated layer by layer through the multi-stage adder tree to form an intermediate result of the accumulated sum, and then sent to the accumulator. Finally, the accumulator calculates the accumulated result through shifting and addition operation. In the next clock cycle, new activation value bits are sent in for calculation and accumulation again until all the input activation values are processed.

[0021] The adaptive gating unit is used to adjust the on-off state of the memory-compute bank in the memory-compute macro unit in different computing stages. According to the current workload, computing mode and clock cycle, idle modules are turned off to reduce power consumption, such as unused memory-compute banks.

[0022] The stage-aware mode control unit dynamically monitors the computing load and the current inference stage, and switches the computing mode according to the load needs.

[0023] The peripheral circuit control unit is responsible for the timing and signals of the entire architecture. The word line driving unit is used to control the word line signals of several memory-computing partitions. According to the address and gating signal sent by the peripheral circuit control unit, the target row word line is driven, so that the corresponding 8T memory-computing unit is selected. The bit line driving unit controls the bit line signals of several memory-computing partitions. In the write operation, the bit line drives the data to be written into the target 8T memory unit; in the read operation, the sense amplifier detects the bit line voltage difference to output the data, and in the CIM mode, the bit line current or voltage change is used to represent the partial product information. The clock unit is responsible for providing the system working clock and the synchronization signal of each stage. The weight input unit is responsible for writing external weight data into each 8T memory-computing unit array, supporting multi-bit weight format, and configuring multiple memory-computing partitions according to the bit width of weight quantization, such as 2b, 4b, 8b, etc. The result output and post-processing unit is used to output the multiplication and addition result, and complete the self-attention calculation related operation.

[0024] Figure 2 The workflow of the pre-filling stage with specific computing mode in the adaptive hybrid computing architecture is demonstrated, such as the Attention mechanism in the Transformer model. This stage is usually at the beginning of generating a sequence, and the core of the pre-filling stage is to quickly process a large number of parallel input information, which belongs to a computationally intensive task. In the architecture of the present application, the computationally intensive task can be offloaded to the memory-computing macro unit for parallel execution in the CIM mode, greatly reducing the data transfer time, thereby improving the efficiency of the pre-filling stage. The main process is as follows:

[0025] S01. Input sequence: the system receives input sequence data, including original text, Embedding vector, etc.

[0026] S02. Calculation of self-attention mechanism: the input sequence is X = [x1, x2,..., xn], each xi (i = 1, 2,..., n) is a word vector. In order to calculate self-attention, three different matrices are needed: query matrix Q, key matrix K and value matrix V. Q = XWq, K = XWk, V = XWv, where Wq, Wk and Wv are learnable weight matrices. After generating Q, K and V, the attention score is calculated by the scaling dot product operation. The attention score measures the correlation between each word and other words.

[0027] S03. Pre-filling stage execution: computationally intensive, 8T memory-computing unit array stores and calculates operations related to Attention mechanism. The CIM mode of the 8T memory-computing unit can efficiently perform matrix multiplication and accumulation operation, storing and processing weight matrix in place, and performing efficient multiplication and accumulation operation with input activation value.

[0028] S04. Multi-head attention layer output: The output matrix fuses the context information and gets the relevance of different information in the context. It allows the model to run the self-attention mechanism in multiple different subspaces simultaneously. Each attention head has its own weight matrix, and each head can focus on different features. The output matrices of all heads are spliced, and an additional linear transformation Wo is used to get the final multi-head attention output.

[0029] S05. Feed-forward network layer performs nonlinear transformation: Each sublayer is followed by a residual connection and layer normalization, and finally, the model outputs a sequence.

[0030] Figure 3 The workflow of the decoding phase for sequence generation tasks in the adaptive hybrid computing architecture is shown. This phase is usually autoregressive, generating tokens of the output sequence one by one. The key-value cache KV Cache is repeatedly accessed, which is a memory-intensive task. In the architecture of the present application, memory-intensive tasks can be offloaded to the SRAM mode of the computing macrocell, which can improve decoding efficiency and reduce overall power consumption. The main process is as follows:

[0031] S11. Input of newly generated Token and sequence expansion: The KV Cache is used to output Token one by one, and the KV pair calculated by the newly generated Token is used to update the KV Cache.

[0032] S12. Key-value cache update: Q=xiWq, K=XWk, V=XWv, where Wq, Wk and Wv are learnable weight matrices. After generating Q, K and V, the attention score is calculated by the scaling dot product operation. The attention score measures the relevance between each word and other words.

[0033] S13. Decoding phase execution: It is memory-intensive, and the Key and Value vectors of the previous text are needed every time a new Token is calculated. Since the Token is generated one by one, it is not necessary to calculate the vectors of all Tokens every time, but the generated Key and Value can be reused, which can reduce a lot of computational overhead.

[0034] S14. Multi-head attention layer output: The output matrix fuses the context information and gets the relevance of different information in the context, allowing the model to run the self-attention mechanism in multiple different subspaces simultaneously. Each attention head has its own weight matrix, and each head can focus on different features. The output matrices of all heads are spliced, and an additional linear transformation Wo is used to get the final multi-head attention output.

[0035] S15. The feedforward network layer performs a nonlinear transformation: each sub-layer is followed by a residual connection and layer normalization. Finally, if the sequence termination condition is not reached, repeat the process with the newly generated Token as input; if the termination condition is reached, exit the loop and the model outputs the sequence.

[0036] The structure of the 8T storage and calculation unit in the application is as shown in the figure, which is composed of two PMOS tubes and six NMOS tubes. Figure 4

[0037] The source of the first PMOS tube PMOS0 is connected to VDD, the drain of the first PMOS tube PMOS0 is connected to the storage node Q, and the gate of the first PMOS tube PMOS0 is connected to the storage node QB.

[0038] The source of the second PMOS tube PMOS1 is connected to VDD, the drain of the second PMOS tube PMOS1 is connected to the storage node QB, and the gate of the second PMOS tube PMOS1 is connected to the storage node Q.

[0039] The source of the first NMOS tube NMOS0 is connected to ground, the drain of the first NMOS tube NMOS0 is connected to the storage node Q, and the gate of the first NMOS tube NMOS0 is connected to the storage node QB.

[0040] The source of the second NMOS tube NMOS1 is connected to ground, the drain of the second NMOS tube NMOS1 is connected to the storage node QB, and the gate of the second NMOS tube NMOS1 is connected to the storage node Q.

[0041] The source of the third NMOS tube NMOS2 is connected to the storage node Q, the drain of the third NMOS tube NMOS2 is connected to the bit line BL, and the gate of the third NMOS tube NMOS2 is connected to the word line WL.

[0042] The source of the fourth NMOS tube NMOS3 is connected to the storage node QB, the drain of the fourth NMOS tube NMOS3 is connected to the bit line BLB, and the gate of the fourth NMOS tube NMOS3 is connected to the word line WL.

[0043] The source of the fifth NMOS tube NMOS4 is connected to the drain of the sixth NMOS tube NMOS5 as a product output end, the drain of the fifth NMOS tube NMOS4 is connected to an activation input end, and the gate of the fifth NMOS tube NMOS4 is connected to the storage node Q.

[0044] The source of the sixth NMOS tube NMOS5 is connected to ground, and the gate of the sixth NMOS tube NMOS5 is connected to the storage node QB.

[0045] ​The 8T memory-computing unit in the application combines static random access memory (SRAM) and compute-in-memory (CIM) two computing modes, and has the advantage of no read-write interference. In the SRAM mode, for the read operation, before the WL is enabled, the BL and BLB are pre-charged to VDD. According to the data stored in the Q / QB node, after the WL is enabled, the corresponding bit line is discharged along the storage node through the third NMOS tube NMOS2 / fourth NMOS tube NMOS3, resulting in a voltage drop. The sense amplifier captures the voltage change and outputs the result. For the write operation, the data to be written is loaded to the bit line BL and the bit line BLB, and once the WL is enabled, the bit line is conditionally discharged / charged to the Q / QB node through the third NMOS tube NMOS2 / fourth NMOS tube NMOS3, covering the original stored data, and the write is completed. In the CIM mode, the weight data stored in Q / QB is directly loaded to BL / BLB, instead of traditional read-write. The input signal is activated and applied to the gate of the fifth NMOS tube NMOS4 and the sixth NMOS tube NMOS5, realizing the multiplication operation of the weight and the input, and the matrix multiplication calculation is completed through the transmission transistor logic AND gate. During the calculation process, the WL remains closed, and the third NMOS tube NMOS2 and the fourth NMOS tube NMOS3 are completely closed during the matrix multiplication, and there is no read-write interference problem. The transmission transistor logic AND gate is composed of the fifth NMOS tube NMOS4 and the sixth NMOS tube NMOS5.

[0046] For any 8T memory-computing unit, the logical true value results when performing the multiplication function under different input conditions are as shown in the following table:

[0047]

[0048] It is illustrated that when the stored weight Weight and the input activation value Activation take different combinations, the product output result is weight x activation. When the weight is 0, no matter how the activation value is, the product output is always 0; when the weight is 1, the product output is equal to the activation value.

[0049] The working process of the adaptive gating unit and the stage perception mode control unit is as shown in Figure 5 Figure 1 ​The core component of the dynamic scheduling of the adaptive hybrid computing architecture makes the architecture have the dynamic allocation capability of computing and memory resources. The input layer is responsible for receiving system state monitoring signals such as power consumption, delay, load, storage and computing unit utilization, task stage and the like, and running feedback signals from the previous layer. The decision layer is the core of the control logic, and integrates an adaptive gating unit and a stage-aware mode control unit. The adaptive gating unit dynamically opens or closes the storage and computing partitions in the storage and computing macro unit according to the system load and clock state as needed; when it is detected that part of the storage and computing unit is in an idle or low load state, the gating logic can close the corresponding partition through clock gating or power gating mechanism, thereby realizing fine power consumption management. The stage-aware mode control unit dynamically switches the computing mode according to the inference stage and computing load characteristics, including the computing-intensive, memory-intensive or hybrid mode, to realize the balanced optimization of performance and energy efficiency. The adaptive gating unit is responsible for fine-grained power consumption control of hardware resources, and the stage-aware mode control unit is responsible for macro scheduling of the computing mode; the two work together to generate control decisions by comprehensively evaluating the current system state, task stage and hardware resource availability, and determine the optimal fine-grained configuration mode of each storage and computing partition in the storage and computing macro unit, so as to realize the efficient cooperation of the computing-intensive, memory-intensive and hybrid computing modes. The output layer generates specific control signals according to the decision results, which are used to dynamically adjust the computing mode of the next layer.

[0050] The computing flow of the adaptive hybrid computing architecture in the application is shown in Figure 6 The architecture can dynamically adjust the allocation strategy of computing and memory resources according to the inference stage of the large model, thereby improving the overall computing efficiency and system performance. The specific process is as follows: first, start computing, input the sequence after preprocessing into the hybrid computing architecture. Then, dynamically adjust the computing strategy according to the inference stage Prefill or Decode. When in the Prefill stage, the system enters the computing-intensive mode, and the scheduling module configures more storage and computing units in the storage and computing partition to the CIM working mode to support large-scale matrix operation; when in the Decode stage, the system enters the memory-intensive mode, and the scheduling module configures more storage and computing units in the storage and computing partition to the CIM working mode to improve the energy efficiency and data access efficiency. Finally, the system outputs and processes the computing result, and ends the computing.

[0051] For those skilled in the art, other various corresponding changes and deformations can be made according to the above described technical solutions and concepts, and all these changes and deformations should belong to the protection scope of the claims of the application.

Claims

1. A large model oriented adaptive hybrid computing architecture, characterized in that, Comprise: A peripheral circuit control unit, a word line driving unit, a bit line driving unit, a clock unit, a weight input unit, an adaptive gating unit, a stage perception mode control unit, a memory-compute macro unit, and a result output and post-processing unit; wherein the memory-compute macro unit is composed of a plurality of memory-compute sub-zones, each of which contains an 8T memory-compute unit array, a multi-stage addition tree, and an accumulator; the adaptive gating unit and the stage perception mode control unit cooperatively control the entire memory-compute macro unit to work in an SRAM mode, a CIM mode, or a hybrid computing mode combining the two; In the CIM mode, the activation values are sent from the input port to the 8T memory-compute unit array of each memory-compute sub-zone in a bit serial manner to perform multiply-add computation; each 8T memory-compute unit is used to store weights, and a group of N units is used to represent a multi-bit weight; each group of units completes a bit-level multiplication operation of 1b x Nb to generate a partial product; all partial products are accumulated layer by layer through the multi-stage addition tree to form an intermediate result, which is then sent to the accumulator for accumulation operation; in the next clock cycle, new activation value bits are sent in for computation and accumulation until all input activation values are processed; The stage perception mode control unit dynamically monitors the computation load and the current inference stage, and switches the computation mode according to the load needs; When it is detected that part of the memory-compute units are in an idle or low-load state, the gating logic can turn off the corresponding sub-zone through clock gating or power gating mechanism; the adaptive gating unit dynamically opens or closes the memory-compute sub-zone in the memory-compute macro unit according to the system load and clock state; the stage perception mode control unit dynamically switches the computation mode according to the inference stage and the computation load characteristics; The adaptive gating unit is responsible for fine-grained power control of hardware resources, and the stage perception mode control unit is responsible for macro scheduling of the computation mode.

2. The adaptive hybrid computing architecture for large models of claim 1, wherein, The peripheral circuit control unit is responsible for the timing and signals of the entire architecture; the word line driving unit is used to control the word line signals of a plurality of memory-compute sub-zones, and drives the target row word line according to the address and strobe signals issued by the peripheral circuit control unit, so that the corresponding 8T memory-compute unit is selected; the bit line driving unit controls the bit line signals of a plurality of memory-compute sub-zones, and drives the bit line to write data into the target 8T memory unit during write operation; during read operation, the sense amplifier detects the bit line voltage difference to output data; in the CIM mode, the bit line current or voltage change is used to represent the partial product information; the clock unit is responsible for providing system working clock and phase synchronization signals; the weight input unit is responsible for writing external weight data into each 8T memory-compute unit array; the result output and post-processing unit is used to output multiply-add results and complete self-attention calculation related operations.

3. The adaptive hybrid computing architecture for large models of claim 2, wherein, The weight input unit supports multi-bit weight format, and can configure a plurality of memory-compute sub-zones according to the bit width of weight quantization.

4. The adaptive hybrid computing architecture for large models of claim 3, wherein, The 8T memory-compute unit is composed of two PMOS tubes and six NMOS tubes: The first PMOS tube (PMOS0) source is connected with VDD, the first PMOS tube (PMOS0) drain is connected with storage node Q, and the first PMOS tube (PMOS0) gate is connected with storage node QB; The second PMOS tube (PMOS1) source is connected with VDD, the second PMOS tube (PMOS1) drain is connected with storage node QB, and the second PMOS tube (PMOS1) gate is connected with storage node Q; The first NMOS tube (NMOS0) source is grounded, the first NMOS tube (NMOS0) drain is connected with storage node Q, and the first NMOS tube (NMOS0) gate is connected with storage node QB; The second NMOS tube (NMOS1) source is grounded, the second NMOS tube (NMOS1) drain is connected with storage node QB, and the second NMOS tube (NMOS1) gate is connected with storage node Q; The third NMOS tube (NMOS2) source is connected with storage node Q, the third NMOS tube (NMOS2) drain is connected with bit line BL, and the third NMOS tube (NMOS2) gate is connected with word line WL; The fourth NMOS tube (NMOS3) source is connected with storage node QB, the fourth NMOS tube (NMOS3) drain is connected with bit line BLB, and the fourth NMOS tube (NMOS3) gate is connected with word line WL; The fifth NMOS tube (NMOS4) source is connected with the drain of the sixth NMOS tube (NMOS5) as a product output end, the fifth NMOS tube (NMOS4) drain is as an activation value input end, and the fifth NMOS tube (NMOS4) gate is connected with storage node Q; The sixth NMOS tube (NMOS5) source is grounded, and the sixth NMOS tube (NMOS5) gate is connected with storage node QB.

5. The adaptive hybrid computing architecture for large models of claim 4, wherein, In the CIM mode, the calculation process word line WL remains closed, so that the third NMOS tube (NMOS2) and the fourth NMOS tube (NMOS3) are completely closed during matrix multiplication.

Images