A method for CAM gated in-memory computation based on 3D NAND

Abstract

This invention provides a method for implementing CAM (Content Addressing Memory) gating in-memory computation based on 3D NAND, belonging to the field of novel storage and computing technology. This invention utilizes the inherent three-dimensional string structure of 3D NAND memory to integrate the Content Addressing Memory (CAM) unit for dynamic expert selection and the in-memory computation (CIM) unit for expert computation on the same 3D NAND string. The search results of CAM are directly used as gating signals to gating the computation results of CIM in situ. This allows for the fusion of dynamic expert selection and expert activation vector matrix multiplication calculations within a single computation cycle, eliminating redundant computations and significantly improving computational parallelism and energy efficiency. This invention is applicable to various types of 3D NAND memory technology.

Description

Technical Field

[0001] This invention relates to the field of novel storage and computing technology, specifically to a method for implementing CAM gated in-memory computing based on 3D NAND. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) based on the Transformer architecture have achieved breakthroughs in tasks such as natural language processing, dialogue systems, code generation, and multimodal reasoning. The continuous improvement of the capabilities of large language models mainly relies on the ever-expanding scale of model parameters, leading to a dramatic increase in computational cost. To address this issue, researchers proposed Mixture-of-Experts (MoE) models. By replacing the dense feed-forward network (FFN) layers with multiple parallel expert subnetworks, and having a router dynamically and sparsely activate a small number of experts for each input token, the model parameters can be scaled to trillions while maintaining low computational overhead. The MoE model has become the mainstream architecture for the most advanced large language models in the industry.

[0003] However, edge deployment of MoE models faces significant challenges. Although only a few experts are activated during each MoE inference iteration, the parameters of all experts need to be stored, resulting in substantial storage overhead. 3D NAND flash memory, with its vertical stacking of hundreds of layers, can achieve high-density on-chip storage and is a strong candidate technology for on-chip deployment of large models like MoE. In addition to traditional 3D NAND Flash based on floating gate (FG) devices, new 3D NAND technologies such as ferroelectric 3D NAND (3D FeNAND), charge-trapping 3D NAND, and oxide-channel 3D NAND have also demonstrated high-density and high-energy-efficiency potential in recent years, providing diverse implementation paths for the storage and computation of large MoE models.

[0004] Currently, academia and industry have proposed building a compute-in-memory (CIM) architecture based on 3D NAND memory, which directly performs vector-matrix multiplication operations within the storage array to reduce data movement. However, existing 3D NAND CIM architectures are mainly designed for dense computational workloads and suffer from severe efficiency bottlenecks when handling the dynamic sparse computational workloads of MoE models. Because each layer of 3D NAND is a rectangular structure with a large aspect ratio, multiple parallel experts need to be continuously mapped and stored along the page direction, and input signals need to be broadcast to all experts in the same layer. This results in a large amount of redundant CIM computation being performed on inactive experts, reducing the computational energy efficiency of 3D NAND CIM for MoE inference.

[0005] Therefore, how to combine the unique 3D vertical string structure characteristics of 3D NAND memory with the features of CIM architecture to achieve the integration of dynamic expert selection and expert computation, and eliminate redundant computation, improve computational parallelism and energy efficiency at the hardware level, is a key problem that needs to be solved to realize efficient inference at the edge of MoE large language models. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention proposes a method for CAM gated in-memory computation based on 3D NAND.

[0007] The technical solution of the present invention is as follows:

[0008] A method for CAM gated in-memory computation based on 3D NAND, characterized in that the 3D NAND comprises several blocks and their peripheral circuits, each block consisting of multiple 3D NAND strings, wherein the 3D NAND strings are memory devices stacked in series vertically, the top of each 3D NAND string is connected to the bit line BL via a string select transistor SSL, and the bottom of each 3D NAND string is connected to the common source line SL via a ground select transistor GSL. The specific computation steps include:

[0009] 1) The 3D NAND string is divided into upper and lower parts. The upper part consists of several CAM units (Content Addressable Memory, CAM), each CAM unit is composed of two adjacent memory devices. The lower part consists of several CIM units, each CIM unit is a memory device. All CAM units on the same physical layer in the block constitute a CAM layer, and all CIM units on the same physical layer constitute a CIM layer. All memory devices in each CIM layer or CAM layer share control in the horizontal direction through word lines WL. The CAM layer is used to store the unique identifier of the MoE expert; the CIM layer is used to store the weight parameters of the expert and implement the multiplication operation between input activation and weight.

[0010] 2) During the calculation phase, input activation is achieved by applying the voltage of the modulated SL to the CIM unit. When the MoE router dynamically selects the identifier of a certain expert, the identifier is broadcast as a query signal to the WL corresponding to the CAM unit of all blocks. All 3D NAND strings perform search matching in parallel. Only the CIM unit on the 3D NAND string corresponding to the selected expert outputs the multiplication result and accumulates the result on BL. Thus, the vector matrix multiplication operation of all activated experts is completed in parallel within a single calculation cycle.

[0011] Furthermore, the expert-unique identifier for the MoE model is derived from the CAM cells in the interleaved mapping CAM layer.

[0012] Furthermore, the expert's weight parameters are stored interleaved along the BL direction in the CIM cells of different 3D NAND strings in the same layer of the 3D NAND.

[0013] Furthermore, the storage device of the 3D NAND memory is not limited to traditional floating gate flash memory devices. Ferroelectric NAND (FeNAND), charge-trapping NAND, NAND devices using oxide channels (such as indium gallium zinc oxide such as IGZO), and any other three-dimensional stacked string structure storage technology that supports multi-threshold voltage storage can all serve as the physical carrier of the 3D NAND memory described in this invention.

[0014] Furthermore, the MoE model includes a standard MoE model that selects k experts from N experts, and a grouped MoE model that selects one expert from each of the k expert groups. It utilizes two equivalent methods to extend the representational capability of the CAM unit to achieve finer-grained expert selection: one method is to extend the bit width of a single CAM unit by utilizing the multi-value storage capability of the storage device; the other method is to configure multiple layers of CAM units in series on the same 3D NAND string.

[0015] The technical effects of this invention are as follows:

[0016] 1. This invention integrates the CAM unit and the CIM unit in situ on the same 3D NAND string. By utilizing the inherent series current characteristics of the 3D NAND string, the CAM matching result is directly used as the current gating signal of the CIM calculation path. This enables the dynamic selection and activation of experts for MoE to be realized in a single calculation cycle, thus completely eliminating the redundant calculation overhead of inactive experts from the hardware level.

[0017] 2. This invention combines the staggered expert mapping strategy with the CAM selection mechanism, which fully utilizes the high storage density of 3D NAND while significantly improving the effective computational parallelism during MoE model inference, enabling multiple activated experts to complete calculations in parallel within the same cycle.

[0018] 3. By extending the bit width of CAM units, this invention can flexibly support fine-grained selection of any number of experts, has good scalability, and can adapt to various MoE model variants such as standard MoE and grouped MoE.

[0019] 4. This invention has good versatility for underlying 3D NAND memory devices. It can be implemented based on traditional floating gate Flash devices, as well as on various new three-dimensional stacked memory technologies such as ferroelectric NAND, charge-trap NAND, and oxide channel NAND. It has broad process adaptability and provides a feasible solution for efficient inference of large language models at the edge MoE. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the three-dimensional string structure of 3D NAND in a specific embodiment of the present invention, showing the vertical integration relationship between the CAM unit and the CIM unit on the same 3D NAND string;

[0021] Figure 2 This is an overall schematic diagram of a specific embodiment of the present invention and a working schematic diagram of its application to expert interleaving mapping and selection in the MoE model;

[0022] Figure 3 This is a schematic diagram illustrating the encoding and matching principle of a 2-bit CAM unit in a specific embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram illustrating the encoding and calculation principle of a 1-bit CIM unit in a specific embodiment of the present invention;

[0024] Figure 5 This is the truth table of the final calculation results in a specific embodiment of the present invention;

[0025] Figure 6This is a schematic diagram of a structure in a specific embodiment of the present invention that achieves finer-grained expert selection by connecting multiple CAM units in series. Detailed Implementation

[0026] The present invention will be further clearly and completely described below with reference to the accompanying drawings and specific embodiments.

[0027] The 3D NAND proposed in this invention, such as Figure 1 As shown, 3D NAND consists of multiple blocks, each containing a string of 3D NAND strings composed of numerous vertically stacked memory devices. Each 3D NAND string is connected to the BL (Block Line) at the top via an SSL (Secure Line Line) and to the common source line SL (Gate Line Line) at the bottom via a GSL (Gate Line Line). All memory devices on the same physical layer share the same WL (Wide Line Line). This invention divides each 3D NAND string vertically into two parts: the upper part contains several CAM (Construction Machine) units, and the lower part contains several CIM (Construction Information Model) units. The CAM layer stores the unique identifier of the MoE (Motion Evaluator) expert; the CIM layer stores the expert's weight parameters. Specifically,

[0028] A CAM cell consists of two memory cells connected vertically in series along a 3D NAND string, by programming the two memory cells to two or more complementary threshold voltages (V). TH The CAM unit stores one or more bits of expert identifier information in its state. Correspondingly, during the search phase, a gate voltage corresponding to the query is applied to the WL corresponding to these CAM units. The matching result of the CAM unit is reflected by the on / off characteristics of the 3D NAND string itself: when the CAM unit matches, the 3D NAND string is turned on, outputting a high current, and the selection signal M is 1; when the CAM unit does not match, the 3D NAND string is turned off, outputting a low current, and the selection signal M is 0. Due to the series current limiting characteristics of the 3D NAND string and the logic selection function implemented by the CAM unit, the multi-value characteristics of the 3D NAND storage unit can be effectively utilized to realize a compact multi-bit CAM unit.

[0029] A CIM cell consists of a memory cell vertically connected in series along a 3D NAND string. Weighting parameters (W) are stored by programming the memory cells to different threshold voltage states. (High V) TH This indicates that W is 0 and V is low. THW is represented as 1; during the calculation phase, the input activation (x) is applied to the CIM unit by modulating the voltage of SL, and the CIM unit outputs the multiplication result as (x·W); since the CAM unit and the CIM unit are located on the same 3D NAND string and are connected in series, the current finally read on BL is the logical AND result of M and (x·W), that is, the final output result y = M·(x·W), so that only the CIM operation result on the 3D NAND string selected by the CAM unit contributes to the final output current, that is, only the activated experts are calculated, thereby in-situ gating and eliminating redundant calculations on inactive experts at the hardware level.

[0030] like Figure 2 As shown, this embodiment uses a MoE model that dynamically activates one expert from four experts as an example. The four experts are each assigned a 2-bit binary identifier entry, namely "00", "01", "10", and "11", and stored in the 2-bit CAM cell at the top of their corresponding 3D NAND string. The weight parameters of the four experts are interleaved along the BL direction and stored alternately in the CIM cells of different 3D NAND strings in the same layer of the 3D NAND using an interleaved mapping strategy. When the MoE router dynamically selects expert 0 in a certain inference cycle, its corresponding 2-bit entry "00" is broadcast as a query signal "query 00" to the two WLs corresponding to the CAM layer of all 3D NAND blocks; simultaneously, the input activation signal x for this cycle is applied to the CIM layer through SL. Only the CAM returns a match for the portion of the 3D NAND string corresponding to expert 0 with entry "00", and its CIM calculation result is accumulated onto the BL in the form of current. However, for all 3D NAND strings with entries "01", "10", and "11", the entire string is truncated due to at least one cutoff device in the CAM layer, and the CIM calculation results on these strings cannot contribute to the BL current. This achieves the effect of only the activated expert 0 participating in the calculation at the hardware level, while the inactive experts 1-3 are automatically masked. In the next cycle, if the MoE router selects another expert, it only needs to switch the query to the corresponding expert identifier input to the CAM layer, while the rest of the circuit remains unchanged, thus completing dynamic sparse MoE inference in a zero-redundancy manner.

[0031] The specific operation steps in this embodiment include:

[0032] 1) Construction and Search of CAM Cells: In the CAM layer above the 3D NAND string, taking a 2-bit CAM cell as an example (e.g.) Figure 3 As shown), it consists of two memory devices connected vertically in series along a 3D NAND string, and the threshold voltage (V) of the two devices is programmed. TH_1 V TH_2A 2-bit expert identifier entry (E) can be encoded. During the search phase, the gate voltage (V) corresponding to the search query (Q) is applied to the WL corresponding to the two memory devices. S1 V S2 When E and Q are perfectly matched, both series-connected memory devices are simultaneously on. In this case, the current flowing through the entire 3D NAND string is equal to the normal CIM on-state current, indicating a match M = 1. When E and Q are not perfectly matched, at least one device will be off due to its gate voltage falling below its threshold voltage, thus truncating the entire 3D NAND string. The current flowing through this off-state is the cutoff current, indicating a mismatch M = 0. This matching operation is essentially equivalent to a bitwise XNOR operation between E and Q. By utilizing the multi-value storage capability of memory devices, a single physical memory layer can implement multi-bit CAM functionality, significantly reducing the number of 3D NAND layers required for CAM.

[0033] 2) Construction and Calculation of CIM Cells: In the CIM layer at the bottom of the 3D NAND string, taking a 1-bit CIM cell as an example (e.g.) Figure 4 As shown), it consists of a storage device, whose threshold voltage state V TH1 or V TH0 The value of the stored weight W is either 0 or 1. During the computation phase, when the input x is 0, V... SL When the input x is 0, there is no conducting current in the 3D NAND string; when the input x is 1, V SL When the voltage is raised to the read voltage, if the stored W is 1, the 3D NAND string is turned on; if the stored W is 0, the 3D NAND string is turned off, thereby realizing the local multiplication operation x·W between the input and the weight.

[0034] 3) In-situ fusion and gating of CAM and CIM: Since the CAM layer and CIM layer are located on the same 3D NAND string and are naturally connected in series, the current sensed on the BL is simultaneously modulated by the CAM matching result M and the CIM calculation result (x·W), and its equivalent expression is y = M·(x·W). Figure 5 As shown, the calculation results of the CIM layer can only be transmitted to the BL through the string when the CAM layer is matched (M = 1). Conversely, for any unmatched 3D NAND string (M = 0), regardless of the calculation results of its CIM layer, the string is truncated as a whole, the CIM results are gated in-situ by the hardware, and its contribution to the BL current is zero. This mechanism naturally shields the calculations of all inactive experts in the MoE model from the hardware layer, eliminates redundant CIM calculations, and significantly improves the effective computational parallelism and energy efficiency.

[0035] This invention achieves fine-grained expert mapping extension: For MoE models with a large number of experts, this invention can extend the representational capability of CAM units in two equivalent ways to achieve finer-grained expert selection: one way is to extend the bit width of a single CAM unit by utilizing the multi-value storage capability of storage devices; the other way is to configure multiple layers of CAM units in series on the same 3D NAND string. Figure 6 As shown, taking the 8-to-1 MoE model as an example, two layers of CAM units can be configured on top of the 3D NAND string, including a 1-bit CAM unit and a 2-bit CAM unit. The 3D NAND string is only fully connected when the storage identifiers of the two CAM units match the query simultaneously. This multi-level cascaded CAM structure provides a wealth of options for flexible mapping of the MoE expert pool.

[0036] This embodiment fully and in detail illustrates the CAM-gated CIM architecture based on 3D NAND and its method for MoE large language model inference. By integrating CAM and CIM functions in situ onto the same 3D NAND string, this invention fully leverages the inherent three-dimensional vertical stacking characteristics of 3D NAND memory. It fuses and completes the dynamic expert selection and activation calculation for MoE within a single computation cycle, eliminating redundant calculations for inactive experts from the hardware level. This provides a novel hardware solution for efficient MoE large language model inference at the edge.

[0037] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A method for implementing CAM gated in-memory computation based on 3D NAND, characterized in that, The 3D NAND includes several blocks and their peripheral circuits. Each block consists of multiple 3D NAND strings, which are vertically stacked memory devices. The top of each 3D NAND string is connected to the bit line BL via a string select transistor SSL, and the bottom of each 3D NAND string is connected to the common source line SL via a ground select transistor GSL. The specific calculation steps include: 1) The 3D NAND string is divided into upper and lower parts. The upper part consists of several CAM units, each CAM unit is composed of two adjacent memory devices. The lower part consists of several CIM units, each CIM unit is a memory device. All CAM units on the same physical layer in the block constitute a CAM layer, and all CIM units on the same physical layer constitute a CIM layer. All memory devices in each CIM layer or CAM layer share control in the horizontal direction through word lines WL. The CAM layer is used to store the unique identifier of the MoE expert; the CIM layer is used to store the weight parameters of the expert and implement the multiplication operation between input activation and weight. 2) During the calculation phase, input activation is achieved by applying the voltage of the modulated SL to the CIM unit. When the MoE router dynamically selects the identifier of a certain expert, the identifier is broadcast as a query signal to the WL corresponding to the CAM unit of all blocks. All 3D NAND strings perform search matching in parallel. Only the CIM unit on the 3D NAND string corresponding to the selected expert outputs the multiplication result and accumulates the result on BL. Thus, the vector matrix multiplication operation of all activated experts is completed in parallel within a single calculation cycle.

2. The method for CAM gated in-memory computation based on 3D NAND as described in claim 1, characterized in that, The unique identifier for the expert in the MoE model is taken from the CAM cell in the interleaved mapping CAM layer.

3. The method for CAM gated in-memory computation based on 3D NAND as described in claim 1, characterized in that, The weight parameters of the experts are stored alternately along the BL direction in the CIM cells of different 3D NAND strings in the same layer of the 3D NAND.

4. The method for CAM gated in-memory computation based on 3D NAND as described in claim 1, characterized in that, The storage devices include floating-gate flash memory devices, ferroelectric NAND, charge-trapping NAND, and NAND devices using oxide channels.

5. The method for CAM gated in-memory computation based on 3D NAND as described in claim 1, characterized in that, The MoE model adopts the standard MoE model that selects k experts from N experts, and the grouped MoE model that selects one expert from each of the k expert groups.

Images