Three-dimensional integrated store-compute system, model inference method, and electronic device

Abstract

The embodiment of the present disclosure discloses a three-dimensional integrated storage and calculation system, a model reasoning method and an electronic device, wherein the system comprises: a multi-layer chip comprising at least one storage chip and at least one calculation chip; the multi-layer chip is integrated in a three-dimensional stacking manner in the vertical direction; the components in the multi-layer chip comprise a first function acceleration processor and a second function acceleration processor; the first function acceleration processor comprises a first storage component and a first calculation component; the first calculation component is configured to perform a matrix multiplication calculation task through a storage and calculation integrated array in the first calculation component; the second function acceleration processor comprises a second storage component and a second calculation component; the second storage component is used to store data required by the second calculation component to perform a nonlinear operator calculation task; and the second calculation component is configured to perform the nonlinear operator calculation task through a nonlinear acceleration subcomponent in the second calculation component. The present disclosure can improve the computing power and bandwidth of the calculation hardware.

Description

Technical Field

[0001] This disclosure relates to the field of semiconductor technology, and in particular to a three-dimensional integrated memory computing system, a model reasoning method, and an electronic device. Background Technology

[0002] With the rapid development of technologies such as Large Language Models (LLMs) and Artificial Intelligence (AI), the computational demands for model inference are growing exponentially. Large models centered around the Transformer architecture, such as the Generative Pre-trained Transformer (GPT) series and the Large Language Model Meta AI (LLaMA) series, involve massive matrix multiplication operations and nonlinear operator operations during their inference process. Matrix multiplication operations are primarily concentrated in attention mechanisms and feed-forward networks (FFNs), while nonlinear operators include softmax operators, layer normalization operators, and the GELU activation function. These matrix multiplication and nonlinear operator operations place extremely high demands on the computing power and bandwidth of chip hardware. Therefore, effectively improving the computing power and bandwidth of computing hardware has become an urgent technical problem to be solved. Summary of the Invention

[0003] Embodiments of this disclosure provide a three-dimensional integrated in-memory computing system, a model inference method, and an electronic device to improve the computing power and bandwidth of the computing hardware of the in-memory computing system.

[0004] According to one aspect of the present disclosure, a three-dimensional integrated in-memory computing system is provided, comprising: A multi-layer chip, the multi-layer chip comprising at least one layer of memory chip and at least one layer of computing chip; The multi-layer chip is integrated using a three-dimensional stacking method in the vertical direction; The components in the multi-layer chip include a first functional acceleration processor and a second functional acceleration processor; The first functional acceleration processor includes a first storage component in the storage components of the at least one layer of storage chip, and a first computing component in the computing components of the at least one layer of computing chip; The first storage component is configured to store the data required by the first computing component to perform matrix multiplication calculation tasks; The first computing component is configured to perform matrix multiplication calculation tasks via the in-memory array in the first computing component; The second functional acceleration processor includes a second storage component in the storage components of the at least one layer of storage chip, and a second computing component in the computing components of the at least one layer of computing chip; The second storage component is configured to store the data required by the second computing component to perform nonlinear operator computation tasks; The second computing component is configured to perform nonlinear operator computation tasks through a nonlinear acceleration subcomponent in the second computing component.

[0005] In some optional embodiments, the first computing component includes: a memory computing control subcomponent and a memory computing integrated computing subcomponent; The memory computing control subcomponent is configured as follows: The working mode of the in-memory computing subcomponent is controlled to be storage mode. According to the weight data loading instruction, the weight data participating in the matrix multiplication calculation task is read from the first storage component and the weight data is written into the in-memory computing subcomponent. The working mode of the in-memory computing sub-component is updated from storage mode to computing mode. It obtains a matrix multiplication calculation task from the main device or the second computing component, and controls the in-memory computing sub-component to execute the matrix multiplication calculation task to obtain the matrix multiplication operation result.

[0006] In some optional embodiments, the in-memory computing sub-component includes: an input driving unit, an in-memory computing array, an addition unit, and an accumulation unit; The in-memory computing control subcomponent is specifically configured to: determine the first input data of the in-memory computing subcomponent according to the matrix multiplication calculation task, and transmit the first input data to the input driving unit of the in-memory computing subcomponent; The input driving unit drives the first input data to the in-memory computing array; The in-memory computing array performs a multiplication operation based on the data from the input driving unit and the weight data pre-written in the in-memory computing array to obtain the multiplication result. The addition unit performs an addition operation on the result of the multiplication operation to obtain the current part and the result; In response to the fact that the current partial sum and result is not the partial sum and result of the last cycle, the accumulation unit performs an addition operation on the current partial sum and result and the previous accumulation result to determine the current accumulation result; or, in response to the fact that the current partial sum and result is the partial sum and result of the last cycle, the current accumulation result is used as the result of the matrix multiplication operation; the previous accumulation result is the accumulation result obtained by the accumulation unit in the previous addition operation.

[0007] In some optional embodiments, the in-memory control subcomponent includes: The mode control register is configured to store the operating mode of the in-memory computing sub-component; The timing control unit is configured to: read the working mode from the mode control register, and control the working timing of the in-memory computing sub-component according to the working mode; The data stream scheduling unit is configured to determine the address signal based on the working mode and the working timing. The address decoding unit is configured to convert the address signal into a word line selection signal in the in-memory computing subcomponent according to the operating mode and the operating timing.

[0008] In some optional embodiments, the first computing component further includes: The first three-dimensional interconnect interface is used for vertical data transmission between multi-layer chips.

[0009] In some optional embodiments, the second computing component includes: The nonlinear acceleration subcomponent is configured to: acquire data to be calculated, perform nonlinear operator calculation tasks based on the data to be calculated, and obtain nonlinear operation results.

[0010] In some optional embodiments, the nonlinear acceleration sub-component includes a variety of nonlinear acceleration units and vector operation units; Any of the nonlinear acceleration units is configured to: acquire first data to be computed by the nonlinear acceleration unit, execute a nonlinear operator computation task corresponding to the nonlinear acceleration unit based on the first data to be computed, and obtain a nonlinear operation result corresponding to the nonlinear acceleration unit. The vector operation unit is configured to acquire second data to be calculated, perform vector operations based on the second data to be calculated, and obtain the vector operation result.

[0011] In some optional embodiments, the second computing component further includes: The external device input / output unit is configured to receive computing instructions from an external master device; The control unit is configured as follows: Obtain the calculation instructions from the external device input / output unit; Based on the computation instructions, determine the matrix multiplication computation task and / or the nonlinear operator computation task; The matrix multiplication calculation task is sent to the first computing component of the first functional acceleration processor, so that the first computing component executes the matrix multiplication calculation task; and / or, the nonlinear acceleration subcomponent is controlled to execute the nonlinear operator calculation task.

[0012] In some optional embodiments, the second computing component further includes: a second three-dimensional interconnect interface; The control unit is specifically configured to send the matrix multiplication calculation task to the first computing component via the second three-dimensional interconnect interface.

[0013] In some optional embodiments, the external device input / output unit is further configured to receive weight data from the master device; The control unit is also configured to write the weight data from the external device input / output unit into the first storage component.

[0014] In some optional embodiments, the first computing component includes: a memory computing control subcomponent and a memory computing integrated computing subcomponent; The second computing component includes: various nonlinear acceleration units, vector operation units, control units, and external device input / output units; The external device input / output unit is configured to receive weight data from a preset model of an external master device after the system is powered on. The control unit is configured to write the weight data into the first storage component; The in-memory computing control subcomponent is configured to: in response to the weight data loading instruction of the control unit, read the weight data from the first storage component and write the weight data into the in-memory computing subcomponent; The control unit is further configured to: acquire calculation instructions from the preset model of the main device, and based on the calculation instructions, control the in-memory computing control subcomponent, the in-memory computing integrated subcomponent, the various nonlinear acceleration units and the vector operation unit to work together to execute the inference calculation task of the preset model.

[0015] In some optional embodiments, the various nonlinear acceleration units include a variety of the following acceleration units: The normalized exponent acceleration unit is configured to perform normalized exponent operations on the data to be computed input to the acceleration unit. The layer normalization acceleration unit is configured to perform layer normalization operations on the data to be computed input to the acceleration unit; The root mean square normalization acceleration unit is configured to perform root mean square normalization on the data to be computed input to the acceleration unit. The activation function acceleration unit is configured to perform a preset activation function operation on the data to be computed input to the acceleration unit.

[0016] According to another aspect of the present disclosure, a model reasoning method based on a three-dimensional integrated in-memory computing system is provided, comprising: The following reasoning process is performed by the three-dimensional integrated in-memory computing system described in any of the above embodiments: Obtain the weight data of the preset model and write the weight data of the preset model into the in-memory computing array in the first computing component; The system acquires input data from the preset model and controls the first computing component and the second computing component to work together to perform inference calculation tasks on the preset model based on the input data. During the inference process, the system executes matrix multiplication calculation tasks in the preset model through the in-memory computing array in the first computing component and executes nonlinear operator calculation tasks in the preset model through the nonlinear acceleration subcomponent in the second computing component.

[0017] In some optional embodiments, controlling the first computing component and the second computing component to work together based on the input data to perform inference calculation tasks on the preset model includes: The input data is driven to the in-memory computing array through the in-memory computing control sub-component in the first computing component; Matrix multiplication is performed using the in-memory array to obtain query vector, key vector, and value vector; The attention weights are obtained by performing matrix multiplication based on the query vector and the key vector through the vector operation unit in the second computing component. The nonlinear acceleration subcomponent in the second computing component determines the attention operation result based on the attention weights and the value vector, and determines the output result of the current attention layer of the preset model based on the attention operation result. In response to the fact that the current attention layer is not the last attention layer, the output of the current attention layer is used as the input data, and the following steps are iteratively executed: The input data is driven to the in-memory computing array through the in-memory computing control sub-component in the first computing component; Matrix multiplication is performed using the in-memory array to obtain query vector, key vector, and value vector; The attention weights are obtained by performing matrix multiplication based on the query vector and the key vector through the vector operation unit in the second computing component. The nonlinear acceleration subcomponent in the second computing component determines the attention operation result based on the attention weights and the value vector, and determines the output result of the current attention layer of the preset model based on the attention operation result. In response to the current attention layer being the last attention layer, the output of the current attention layer is determined as the inference result of the preset model.

[0018] According to another aspect of the present disclosure, an electronic device is provided, comprising: the three-dimensional integrated memory computing system provided in any of the above embodiments.

[0019] In some embodiments, the electronic device further includes: a host device configured to: The application program running the preset model generates the computation task of the three-dimensional integrated in-memory computing system, calls the three-dimensional integrated in-memory computing system to execute the computation task, and completes the inference of the preset model.

[0020] The three-dimensional integrated in-memory computing system, model inference method, and electronic device provided in the above embodiments of this disclosure integrate multiple layers of chips in a vertical three-dimensional stacking manner to achieve a high-density, low-power three-dimensional interconnection in-memory computing system for storage and computing units. This effectively improves the bandwidth and computing power of the in-memory computing system. Furthermore, the components in the three-dimensional integrated in-memory computing system of this disclosure include a first functional accelerator processor and a second functional accelerator processor, which respectively implement matrix multiplication operations and nonlinear operator operations, realizing heterogeneous integration of matrix processing and nonlinear processing. This is beneficial for accelerating end-to-end high-energy-efficiency inference of large models. In addition, the first computing component implements matrix multiplication operations through an integrated in-memory computing array, which can effectively reduce data transfer between storage chips and computing chips, and helps to improve the energy efficiency ratio and throughput of computing hardware.

[0021] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 This is an exemplary application scenario of an embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a three-dimensional integrated in-memory computing system provided in an exemplary embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of a first computing component provided in an exemplary embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of an in-memory computing sub-component provided in an exemplary embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of a memory computing control sub-component provided in an exemplary embodiment of this disclosure; Figure 6 This is a schematic diagram of the structure of a first computing component provided in another exemplary embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of the second computing component provided in an exemplary embodiment of this disclosure; Figure 8 This is a schematic diagram of the structure of a nonlinear accelerator sub-component provided in an exemplary embodiment of this disclosure; Figure 9 This is a schematic diagram of the structure of a second computing component provided in another exemplary embodiment of this disclosure; Figure 10 This is a schematic diagram of the structure of a three-dimensional integrated in-memory computing system provided in another exemplary embodiment of this disclosure; Figure 11 This is a schematic diagram of the structure of a Computation Matrix Processor (CIMMP) provided in an exemplary embodiment of this disclosure; Figure 12 This is a schematic diagram of the structure of a nonlinear neuromorphic processor (NNP) provided in an exemplary embodiment of this disclosure; Figure 13 This is a flowchart of the end-to-end inference workflow of a model provided in an exemplary embodiment of this disclosure; Figure 14 This is a flowchart illustrating a model reasoning method based on a three-dimensional integrated in-memory computing system provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0024] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0025] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0026] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0027] It should also be understood that in the embodiments disclosed herein, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0028] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0029] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0030] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0031] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0032] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0033] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0034] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0035] This disclosure outlines With the rapid development of technologies such as Large Language Models (LLM) and Artificial Intelligence (AI), the computational demands for model inference are growing exponentially. Large models based on the Transformer architecture, such as the Generative Pre-trained Transformer (GPT) series and the Meta Large Language Model (LLaMA) series, involve massive matrix multiplication operations and nonlinear operator operations in their inference process. Among these, matrix multiplication operations are mainly concentrated in attention mechanisms and feed-forward networks (FFNs), while nonlinear operators include softmax operators, layer normalization operators, Gaussian Error Linear Unit (GELU) activation functions, and Sigmoid Linear Unit (SiLU) activation functions. These matrix multiplication operations and nonlinear operator operations place extremely high demands on the computing power and bandwidth of chip hardware. Therefore, how to effectively improve the computing power and bandwidth of computing hardware has become an urgent technical problem to be solved.

[0036] In response to the above technical issues, Figure 1 This is an exemplary application scenario of an embodiment of this disclosure, such as... Figure 1 As shown, the inference application of the preset model can be run through the host device 10. During the inference process, the calculation instructions of the three-dimensional integrated in-memory computing system 20 are generated. The calculation instructions can be used to instruct the three-dimensional integrated in-memory computing system 20 to perform the inference acceleration task of the preset model, thereby realizing the inference acceleration calculation of the preset model through the three-dimensional integrated in-memory computing system 20.

[0037] The main device 10 includes, but is not limited to, one or more of the following: a central processing unit (CPU), a coprocessor, and a controller. The main device 10 can communicate with the 3D integrated in-memory computing system 20 through a preset interface to exchange instructions and data. The preset interface includes, but is not limited to, Peripheral Component Interconnect Express (PCIe) and Compute Express Link (CXL). The preset model may include, but is not limited to, large language models and multimodal AI models.

[0038] The three-dimensional integrated in-memory computing system 20 may include: a multi-layer chip, which includes at least one layer of storage chip and at least one layer of computing chip; the multi-layer chip is integrated in a three-dimensional stacking manner in the vertical direction; the components in the multi-layer chip include a first functional accelerator processor and a second functional accelerator processor; the first functional accelerator processor includes a first storage component in the storage component of at least one layer of storage chip and a first computing component in the computing component of at least one layer of computing chip; the first storage component is configured to: store data required by the first computing component to perform matrix multiplication calculation tasks; the first computing component is configured to: perform matrix multiplication calculation tasks through an in-memory computing array in the first computing component; the second functional accelerator processor includes a second storage component in the storage component of at least one layer of storage chip and a second computing component in the computing component of at least one layer of computing chip; the second storage component is configured to: store data required by the second computing component to perform nonlinear operator calculation tasks; the second computing component is configured to: perform nonlinear operator calculation tasks through a nonlinear acceleration subcomponent in the second computing component.

[0039] The three-dimensional integrated in-memory computing system provided in this disclosure integrates multiple layers of chips in a vertical three-dimensional stacking manner, achieving a high-density, low-power three-dimensional interconnection between storage units and computing units. This effectively improves the bandwidth and computing power of the in-memory computing system. Furthermore, the components in the three-dimensional integrated in-memory computing system of this disclosure include a first functional accelerator and a second functional accelerator, which respectively implement matrix multiplication operations and nonlinear operator operations, realizing heterogeneous integration of matrix processing and nonlinear processing. This is beneficial for accelerating end-to-end high-energy-efficiency inference of large models. In addition, the first computing component implements matrix multiplication operations through an integrated in-memory computing array, which can effectively reduce the number of data transfers between storage chips and computing chips, helping to improve the energy efficiency ratio and throughput of computing hardware.

[0040] Exemplary Structure Figure 2 This is a schematic diagram of the structure of a three-dimensional integrated in-memory computing system provided in an exemplary embodiment of this disclosure. The various components of this three-dimensional integrated in-memory computing system 20 can be integrated into a single element or distributed among multiple independent elements, with data communication links established between these elements. For example... Figure 2 As shown, the three-dimensional integrated in-memory computing system 20 may include: a multi-layer chip, which includes at least one layer of memory chip and at least one layer of computing chip. The multi-layer chip is integrated in a three-dimensional stacking manner in the vertical direction.

[0041] The components in the multi-layer chip include a first functional accelerator 21 and a second functional accelerator 22.

[0042] The first functional acceleration processor 21 includes a first storage component 211 in a storage component of at least one layer of storage chip, and a first computing component 212 in a computing component of at least one layer of computing chip.

[0043] The first storage component 211 is configured to store the data required by the first computing component 212 to perform matrix multiplication calculation tasks.

[0044] The first computing component 212 is configured to perform matrix multiplication calculation tasks through the in-memory array in the first computing component 212.

[0045] The second functional acceleration processor 22 includes a second storage component 221 in a storage component of at least one layer of storage chip, and a second computing component 222 in a computing component of at least one layer of computing chip.

[0046] The second storage component 221 is configured to store the data required by the second computing component 222 to perform nonlinear operator computing tasks.

[0047] The second computing component 222 is configured to perform nonlinear operator computing tasks through the nonlinear acceleration subcomponent in the second computing component 222.

[0048] In this embodiment of the disclosure, a multi-layer chip can refer to a multi-layer die, or a multi-layer chip. For example, a memory chip can be a memory die, or a memory chip. A computing chip can be a computing die, or a computing chip.

[0049] Optionally, each layer of storage chips may include high-capacity storage components. For example, storage chips may be, but are not limited to, high-bandwidth memory (HBM) chips. Each layer of computing chips may include a computing-in-memory array (CIM array) and other related components. The computing-in-memory array may be, but is not limited to, a high-computing-power array.

[0050] Optionally, the in-memory computing array can be an array in the digital domain or an array in the analog domain, and this disclosure does not limit the specific implementation of the in-memory computing array. In some examples, the analog domain in-memory computing array can perform multiplication and addition operations using analog circuit techniques such as charge sharing and current mirroring. In other examples, the digital domain in-memory computing array can be a two-dimensional array composed of multiple digital domain in-memory computing units. The in-memory computing array is, for example, but not limited to, a 512×512 two-dimensional array, a 1024×1024 two-dimensional array, etc. Each in-memory computing unit can include a memory cell and digital logic gates. Optionally, the memory cell can be of any type. For example, a memory cell may include, but is not limited to, one or more of the following: SRAM Cell (Static Random Access Memory Cell), STT-MRAM (Spin Transfer Torque Magnetic Random Access Memory) Cell, RRAM (Resistive Random Access Memory) Cell, PCM (Phase Change Memory) Cell, and FeFET (Ferroelectric Field Effect Transistor) Cell. Optionally, an SRAM Cell may include, but is not limited to, 6T SRAM Cell, 8T SRAM Cell, 10T SRAM Cell, and dual-port SRAM Cell. 6T, 8T, and 10T represent the total number of transistors integrated in a single SRAM Cell, where T stands for Transistor. For example, a 6T SRAM Cell represents a six-transistor SRAM memory cell. Digital logic gates can include one or more of the following: NOR gate, NAND gate, transmission gate, and XOR gate, to implement multiplication.

[0051] The in-memory computing array can operate in two modes: Memory Mode and Computation Mode (CIM Mode or Computer Mode). In Memory Mode, read and write operations can be performed on the storage cells of the in-memory computing array. For example, weight data of a neural network model can be pre-written into the storage cells. Optionally, read and write operations can be performed on the storage cells via word lines (WL) and bit lines / bit line bars (BL / BLB). In Computation Mode, input data (or input activation data) can be sent to the in-memory computing unit in the in-memory computing array. Multiplication is performed locally with the pre-stored weight data in the storage cells to obtain the multiplication result. The result is then further processed using an adder tree, accumulator, etc., to obtain the matrix multiplication result. Optionally, input data can be sent to the in-memory computing unit via an input line driver. Optionally, the operation of the in-memory computing array in the corresponding operating mode can be controlled by a controller to realize the writing of weight data, the input of input data, and matrix multiplication operations.

[0052] Multi-layer chips are integrated in a three-dimensional stacking manner in the vertical direction. This means that multi-layer chips are stacked and integrated in the vertical direction and interconnected using a three-dimensional integration method. Compared with the two-dimensional (2D) architecture in the existing technology, three-dimensional integration can significantly shorten the physical distance between storage and computing, greatly improve the interconnect bandwidth density, and reduce signal transmission power consumption.

[0053] Optionally, three-dimensional integration methods include, but are not limited to, through silicon via (TSV) and hybrid bonding, micro-bump and TSV, through chip via (TCV), and 2.5D based on passive silicon interposers (but still retaining the main 3D stacked architecture).

[0054] Alternatively, the hybrid bonding technology in the through-silicon via and hybrid bonding method can be replaced with other direct bonding technologies, such as fusion bonding, anodic bonding, or adhesive bonding.

[0055] In some examples, a multi-layer chip may include an M-layer storage die and an N-layer compute die, where M and N are both positive integers. The M-layer storage die and the N-layer compute die are interconnected in a high-density, low-power three-dimensional manner through a hybrid bonding method of TSV and Hybrid Bonding.

[0056] In some optional embodiments, at least one layer of storage chip and at least one layer of computing chip are physically divided into two functional areas. The first functional area is used to implement the Computing-in-Memory Matrix Processor (CIMMP), i.e., the first functional acceleration processor 21. The first functional acceleration processor 21 can be responsible for accelerating large-scale matrix multiplications in the Attention mechanism, FFN, and other computations. The second functional area is used to implement the Nonlinear Neuromorphic Processor (NNP), i.e., the second functional acceleration processor 22. The second functional acceleration processor 22 can be responsible for accelerating nonlinear operators (such as Softmax, LayerNorm, GELU activation function, SiLU activation function, etc.) in the Attention mechanism. Through this heterogeneous three-dimensional integrated architecture, end-to-end acceleration of large model inference can be achieved. The model's weight data can be stored in the storage chip, and the matrix multiplication operations in the model can be directly completed on the storage side through the in-memory computing array in the first functional acceleration processor 21. Nonlinear operator operations can be efficiently accelerated by the second functional acceleration processor 22. It effectively reduces data transfer between storage and computation during the overall inference process of the model, and effectively improves the system's energy efficiency and throughput.

[0057] In some optional embodiments, a redistribution layer (RDL) can be present between adjacent chip layers to achieve high-density electrical interconnection between adjacent chip layers. The RDL layer and TSV are bonded together by a certain bonding method, thereby combining the RDL layer and TSV to open up the vertical and horizontal signal paths of the multi-layer chip, and to realize the distribution functions of power, network, etc.

[0058] The components in a multi-layer chip include storage components in each layer of memory chips and computing components in each layer of computing chips. The components in a multi-layer chip can be divided into multiple functional areas, not limited to a first functional area and a second functional area. This disclosure only uses two functional areas as an example for illustrative purposes. When the multi-layer chip is physically divided into two functional areas, the storage components of each layer of memory chips may include a first storage component 211 and a second storage component 221. The first storage component 211 is used to store data required by the first functional accelerator 21 to perform acceleration tasks, such as model weight data. The second storage component 221 is used to store data required by the second functional accelerator 22 to perform acceleration tasks, such as data or configuration parameters required to perform nonlinear operator computation tasks. The computing components of each layer of computing chips may include a first computing component 212 and a second computing component 222. The first computing component 212 is used to perform matrix multiplication computation tasks through a memory-computing array, such as, but not limited to, performing matrix multiplication computation tasks in the Attention mechanism, FFN, etc., of the model. The second computing component 222 is used to perform nonlinear operator computing tasks through the nonlinear acceleration subcomponent, such as, but not limited to, performing nonlinear operator computing tasks such as Softmax, LayerNorm, GELU, and SiLU.

[0059] In some optional embodiments, the system of this disclosure may further include a packaging substrate for packaging three-dimensionally integrated multilayer chips. Optionally, the packaging substrate may include a solder ball array, a PCIe / CXL physical layer interface circuit (PHY), a power management integrated circuit (PMIC), etc. The packaging substrate is connected to the bottom chip of the three-dimensional stacked chip through a micro-bump array, and connected to the outside world, such as a PCB (Printed Circuit Board), through a solder ball bump array, to realize the connection between the system of this disclosure and other external devices or components. The PCIe / CXL physical layer interface circuit is used for command and data interaction with the outside world, and the power management module is used to provide power to the components in the system of this disclosure.

[0060] In some optional embodiments, the vertical arrangement of the memory chips and computing chips can be arbitrary and not limited to any particular order. Figure 2The memory chips are placed on the upper layer, and the computing chips on the lower layer. For example, all memory chips could be on the lower layer and the computing chips on the upper layer, or the memory chips and computing chips could be arranged in any mixed manner. For example, but not limited to, a mixed arrangement of one memory chip and one computing chip. Optionally, any layer of chips can be divided into a front and a back side. The mating surfaces of adjacent stacked chip layers can be face-to-back, face-to-face, back-to-back, etc. Taking face-to-back as an example, it means that the upper layer chip is stacked on top of the lower layer chip with its back side facing down, and the lower layer chip has its front side facing up. The upper layer chip can use its TSV (Transient Voltage Segmentation) penetrating its own silicon substrate to guide the front circuit signals to the back side, and then interconnect with the circuitry on the front side of the lower layer.

[0061] In this embodiment of the disclosure, components, sub-components, units, etc., can all be hardware components, and the acceleration of matrix multiplication and nonlinear operations is achieved through hardware.

[0062] The three-dimensional integrated in-memory computing system provided in this disclosure integrates multiple layers of chips in a vertical three-dimensional stacking manner, achieving a high-density, low-power three-dimensional interconnection between storage units and computing units. This effectively improves the bandwidth and computing power of the in-memory computing system. Furthermore, the components in the three-dimensional integrated in-memory computing system of this disclosure include a first functional accelerator and a second functional accelerator, which respectively implement matrix multiplication operations and nonlinear operator operations, realizing heterogeneous integration of matrix processing and nonlinear processing. This is beneficial for accelerating end-to-end high-energy-efficiency inference of large models. In addition, the first computing component implements matrix multiplication operations through an integrated in-memory computing array, which can effectively reduce the number of data transfers between storage chips and computing chips, helping to improve the energy efficiency ratio and throughput of computing hardware.

[0063] Figure 3 This is a schematic diagram of the structure of a first computing component provided in an exemplary embodiment of this disclosure.

[0064] In some alternative embodiments, in the above... Figure 2 Based on the illustrated embodiments, as Figure 3 As shown, the first computing component 212 may include, but is not limited to, a memory computing control subcomponent 2121 and a memory computing integrated computing subcomponent 2122. The memory computing integrated computing subcomponent 2122 is electrically connected to the memory computing control subcomponent 2121.

[0065] The storage control subcomponent 2121 is configured as follows: The working mode of the in-memory computing sub-component 2122 is storage mode. According to the weight data loading instruction, the weight data participating in the matrix multiplication calculation task is read from the first storage component 211 and written into the in-memory computing sub-component 2122. The in-memory computing sub-component 2122 is updated from storage mode to computing mode. It obtains matrix multiplication calculation tasks from the main device 10 or the second computing component 222, and controls the in-memory computing sub-component 2122 to execute the matrix multiplication calculation tasks to obtain the matrix multiplication operation results.

[0066] Among them, the in-memory computing control sub-component 2121 can be called the in-memory computing controller (CIM Controller), which can be used to control the switching of the working mode of the in-memory computing array in the in-memory computing sub-component 2122, address decoding, timing generation, computing flow scheduling, etc.

[0067] The in-memory computing sub-component 2122 is a hardware component for performing matrix multiplication operations. The in-memory computing sub-component 2122 may include an in-memory array and peripheral circuitry of the in-memory array. Peripheral circuitry includes, but is not limited to, input line drivers, word line drivers, adder trees, accumulators, etc.

[0068] Before the compute subcomponent 2122 performs the matrix multiplication calculation task, the in-memory computing control subcomponent 2121 can control the compute subcomponent 2122 to operate in storage mode. Based on the weight data loading instruction, it reads the weight data participating in the matrix multiplication calculation task from the first storage component 211 and writes the weight data into the compute subcomponent 2122. For example, the weight data can be written to the storage unit of the in-memory computing array within the compute subcomponent 2122. The weight data loading instruction can be obtained from the external host device 10, the controller or processor within the system 20, or the second function acceleration processor 22. In some examples, when the host device 10 needs to calculate the matrix multiplication result of two matrices, it can write one of the matrices as weight data into the first storage component 211 and generate a weight data loading instruction. This instruction is then sent to the in-memory computing control subcomponent 2121, which can read the weight data participating in the matrix multiplication calculation task from the first storage component 211 according to the weight data loading instruction and write the weight data into the compute subcomponent 2122. In other examples, the controller or processor within system 20 receives computation instructions from an external master device, parses the instructions, determines the matrix multiplication task, generates a weight data loading instruction based on the task, and sends the instruction to the in-memory control subcomponent 2121. In still other examples, the second functional acceleration processor 22 receives computation instructions from an external master device, parses the instructions, determines the matrix multiplication task, generates a weight data loading instruction, and sends it to the in-memory control subcomponent 2121. These are just some examples of the in-memory control subcomponent 2121 obtaining weight data loading instructions; practical applications are not limited to these examples.

[0069] After the weight data is prewritten, the in-memory computing control subcomponent 2121 can update the working mode of the in-memory computing subcomponent 2122 from storage mode to computing mode, acquire the matrix multiplication calculation task, and control the in-memory computing subcomponent 2122 to execute the matrix multiplication calculation task to obtain the matrix multiplication result. The control of the in-memory computing subcomponent 2122 by the in-memory computing control subcomponent 2121 can include timing control, address decoding, etc. Address decoding provides the storage address of the weight data to the in-memory computing subcomponent 2122, and timing control ensures that the in-memory computing subcomponent 2122 works accurately according to the timing sequence, ensuring the accuracy of the matrix multiplication result.

[0070] Optionally, the matrix multiplication calculation task may include, but is not limited to, parameters such as the storage address of the weight data participating in the matrix multiplication calculation task, the storage address of the input data, and the storage address of the output data. The in-memory computing control subcomponent 2121 can perform address decoding, timing control, and send the input data to the in-memory computing subcomponent 2122 according to the matrix multiplication calculation task. Under the control of the in-memory computing control subcomponent 2121, the in-memory computing subcomponent 2122 completes the matrix multiplication operation between the input data and the pre-stored weight data to obtain the matrix multiplication result.

[0071] Optionally, if the system 20 includes a multi-layer computing chip, the first computing sub-component 212 of the multi-layer computing chip can operate in parallel to complete a large-scale matrix multiplication calculation task.

[0072] In some optional embodiments, the operating mode of the in-memory computing sub-component 2122 can be maintained through an operating mode register. The operating mode register can store flag bits indicating storage mode or computing mode. When the operating mode register stores a flag bit indicating storage mode, the operating mode of the in-memory computing sub-component 2122 is storage mode; when the operating mode register stores a flag bit indicating computing mode, the operating mode of the in-memory computing sub-component 2122 is computing mode.

[0073] In the embodiments of this disclosure, the first computing component controls the working mode of the in-memory computing subcomponent through the in-memory computing control subcomponent, ensuring that the weighted data is pre-written into the storage unit of the in-memory computing array in the storage mode, and then switches to the computing mode to realize matrix multiplication operation, which helps to execute the matrix multiplication calculation task efficiently and accurately.

[0074] Figure 4 This is a schematic diagram of the structure of an in-memory computing sub-component provided in an exemplary embodiment of this disclosure.

[0075] In some alternative embodiments, such as Figure 4 As shown, the in-memory computing sub-component 2122 may include: an input driving unit 21221, an in-memory computing array 21222, an addition unit 21223, and an accumulation unit 21224.

[0076] The in-memory computing control sub-component 2121 is specifically configured to: determine the first input data of the in-memory computing sub-component 2122 according to the matrix multiplication calculation task, and transmit the first input data to the input driving unit 21221 of the in-memory computing sub-component 2122.

[0077] The input driving unit 21221 drives the first input data to the in-memory computing array 21222.

[0078] The in-memory computing array 21222 performs multiplication operations based on data from the input driving unit and weight data pre-written in the in-memory computing array to obtain the multiplication result.

[0079] The addition unit 21223 performs an addition operation on the result of the multiplication operation to obtain the current part and the result.

[0080] In response to the fact that the current partial sum result is not the partial sum result of the last cycle, the accumulation unit 21224 performs an addition operation on the current partial sum result and the previous accumulation result to determine the current accumulation result; or, in response that the current partial sum result is the partial sum result of the last cycle, the current accumulation result is used as the result of the matrix multiplication operation; the previous accumulation result is the accumulation result obtained by the previous addition operation performed by the accumulation unit.

[0081] The input drive unit 21221 can be electrically connected to the in-memory computing control sub-component 2121 and is used to drive input data to the in-memory computing array 21222. Optionally, the input drive unit 21221 may include, but is not limited to, an input data converter, a buffer, and an input line driver. The input data converter is used to convert the data type or format of the input data, such as converting FP16 data to FP8 data. The buffer is used to buffer the input data or the converted input data. The input line driver is used to drive the input data buffered in the buffer to the in-memory computing array 21222. Optionally, the input line driver may, but is not limited to, drive the input data to the in-memory computing array 21222 bit by bit.

[0082] The in-memory computing array 21222 can be electrically connected to the input driving unit 21221. Optionally, each in-memory computing unit in the in-memory computing array 21222 can perform a single-bit multiplication operation. In the current cycle, a single-bit multiplication operation can be performed on the current single-bit data from the input driving unit 21221 and the pre-stored weight data in the storage unit of the in-memory computing array to obtain the current single-bit multiplication result. Optionally, each in-memory computing unit in the in-memory computing array 21222 can perform multiplication operations on multiple bits. The specific implementation of the in-memory computing unit is not limited.

[0083] The addition unit 21223 can be electrically connected to the in-memory computing array 21222. Optionally, the addition unit 21223 can be used to accumulate the multiplication results of multiple in-memory computing units in each column of the in-memory computing array 21222 to obtain the current partial sum result. Optionally, the addition unit 21223 can be, but is not limited to, an adder tree, which performs parallel tree-like accumulation of the outputs of multiple in-memory computing units in the same column to obtain the current partial sum result. Optionally, the adder tree can include a signed number adder tree and an unsigned number adder tree. The signed number adder tree is used to process the sign bit. The unsigned number adder tree is used to process the numerical bits.

[0084] The accumulation unit 21224 can be called an accumulator. Optionally, the accumulation unit 21224 can be electrically connected to the addition unit 21223 to accumulate the cross-cycle parts and results of the addition unit 21223 to obtain the matrix multiplication result. Optionally, the accumulation unit 21224 may include a multi-stage adder and an accumulation register. The multi-stage adder is used to perform the accumulation operation, and the accumulation register is used to store the accumulation result obtained by the accumulation operation. In the current cycle, the accumulation register stores the accumulation result obtained by the previous accumulation operation, called the previous accumulation result. After the accumulation operation of the current cycle is completed, the current accumulation result is overwritten to the accumulation register as the previous accumulation result of the next cycle. Since the input data is bit-driven to the in-memory computing array, it takes multiple cycles to complete the matrix multiplication operation of the input data. In cycles other than the last cycle, the current accumulation result is overwritten to the accumulation register as the previous accumulation result of the next cycle. In the last cycle, the current accumulated result can be used as the result of matrix multiplication.

[0085] Optionally, the accumulator unit 21224 can support accumulation operations with various data precisions, such as INT4, INT8, FP8, FP16, etc., and can be configured with corresponding precision control logic to adapt to the quantization inference requirements of different large models.

[0086] Optionally, the input driving unit 21221, the in-memory computing array 21222, the addition unit 21223 corresponding to each column in the in-memory computing array, and the accumulation unit 21224 can be calculated in a parallel pipeline manner to further improve computing efficiency.

[0087] Optionally, for any in-memory computing sub-component 2122, the matrix multiplication result obtained can be the result of a portion of the matrix multiplication tasks handled by that in-memory computing sub-component 2122. Multi-layer computing components can execute in parallel to complete larger-scale matrix multiplication tasks. For example, the controller or processor within the main device 10 or system 20 can decompose a large-scale matrix multiplication task into smaller matrix multiplication tasks (or matrix multiplication subtasks) that can be supported by each in-memory computing sub-component 2122, and then combine the matrix multiplication results of each in-memory computing sub-component 2122 to obtain the result of the large-scale matrix multiplication task.

[0088] In some optional embodiments, the in-memory computing sub-component 2122 may further include a word line driver. In storage mode, the word line driver can drive word lines to perform read and write operations to pre-write weight data to the storage cells of the in-memory computing array. In computing mode, the word line driver can be used to select target rows in the in-memory computing array.

[0089] In some optional embodiments, the in-memory computing subcomponent 2122 may further include a multibit input transfer logic for receiving multi-cycle partial sums and results, shifting them according to bit weights, and converting the multi-cycle single-bit multiply-accumulate results into multi-bit × multi-bit multiply-accumulate results.

[0090] In the embodiments of this disclosure, an in-memory computing sub-component is implemented through an input driving unit, an in-memory computing array, an addition unit, and an accumulation unit, ensuring that the input data can be accurately driven to the in-memory computing array, which helps to accurately realize matrix multiplication operations.

[0091] Figure 5 This is a schematic diagram of the structure of a memory computing control sub-component provided in an exemplary embodiment of this disclosure.

[0092] In some alternative embodiments, such as Figure 5 As shown, the in-memory control sub-component 2121 may include: a mode control register 21211, a timing control unit 21212, a data stream scheduling unit 21213, and an address decoding unit 21214.

[0093] The mode control register 21211 is configured to operate in the mode of the storage-in-memory computing sub-component 2122.

[0094] The timing control unit 21212 is configured to read the working mode from the mode control register 21211 and control the working timing of the in-memory computing sub-component 2122 according to the working mode.

[0095] The data flow scheduling unit 21213 is configured to determine the address signal based on the working mode and working timing.

[0096] Address decoding unit 21214 is configured to convert address signals into word line selection signals in in-memory computing subcomponent 2122 according to the working mode and working timing.

[0097] The mode control register 21221 is the aforementioned operating mode register. The in-memory computing control subcomponent 2121 can control the operating mode of the in-memory computing subcomponent 2122 by updating the contents stored in the mode control register 21221. Optionally, the mode control register 21221 can also be used to store other relevant calculation parameters. Optionally, the in-memory computing control subcomponent 2121 may include a mode control unit responsible for updating the operating mode stored in the mode control register 21221.

[0098] The timing controller 21212 can be electrically connected to the mode control register 21211. Based on the operating mode read from the mode control register 21211, it controls the operating timing of the in-memory computing sub-component 2122, enabling it to operate in storage mode to perform operations such as pre-writing weight data, or to operate in computing mode to perform matrix multiplication calculations. Different operating modes correspond to different operating timing sequences.

[0099] The dataflow scheduler 21213 can be electrically connected to the mode control register 21211 and the timing control unit 21212. Different operating modes and timing sequences can correspond to different data flows. For example, in storage mode, the data flow may include reading weight data from the first storage component 211 and writing weight data to the in-memory computing subcomponent 2122. In computing mode, the data flow may include inputting data to the computing unit of the in-memory computing array 21222 in the in-memory computing subcomponent 2122 and sending the output data of the in-memory computing array 21222 to the addition unit 21223. Different data flows determine different address signals that need to be decoded. Therefore, the address signals of the data flow to be scheduled can be determined according to the operating mode and timing sequence.

[0100] The address decoder 21214 can be electrically connected to the mode control register 21211, the timing control unit 21212, and the data stream scheduling unit 21213. It can decode the address signal into a word line selection signal in the in-memory computing sub-component 2122 according to the operating mode and timing, thereby selecting the corresponding memory cell to perform multiplication with the input single-bit data.

[0101] Optionally, the storage control sub-component 2121 may also include other related hardware units, not limited to the units mentioned above, and may be configured according to actual needs.

[0102] In the embodiments of this disclosure, the in-memory computing control subcomponent controls the working mode of the in-memory computing subcomponent through a mode control register, controls the working timing of the in-memory computing subcomponent through a timing control unit, determines the address signal of the data stream to be scheduled through a data stream scheduling unit, and then converts the address signal into a word line selection signal of the in-memory computing array in the in-memory computing subcomponent through an address decoding unit, thereby achieving accurate and effective control of the in-memory computing subcomponent and helping to ensure that the in-memory computing subcomponent accurately executes matrix multiplication calculation tasks.

[0103] Figure 6 This is a schematic diagram of the structure of a first computing component provided in another exemplary embodiment of this disclosure.

[0104] In some alternative embodiments, such as Figure 6 As shown, the first computing component 212 may further include: a first three-dimensional interconnect interface 2123 for vertical data transmission between multi-layer chips.

[0105] The first three-dimensional interconnect interface 2123 is a data and control channel between the first functional acceleration processor 21 and the chips of adjacent layers, and can be used for high-speed data transmission between the first functional acceleration processor 21 and other layers.

[0106] Optionally, the first three-dimensional interconnect interface 2123 may be a three-dimensional TSV interface. The first three-dimensional interconnect interface 2123 may include, but is not limited to, a transmitter, a receiver, and bonding interface logic. In some examples, the first three-dimensional interconnect interface 2123 may include a TSV transmitter, a TSV receiver, and hybrid bonding interface logic. The transmitter can be used to send the matrix multiplication result output from the accumulator to other layers via TSV. The receiver can be used to receive input data and configuration instructions from other layers. The hybrid bonding interface logic can be used to manage the physical layer communication protocol with the hybrid bonding pad.

[0107] In this embodiment, high-speed data transmission between layers is achieved through a three-dimensional interconnect interface, enabling multi-layer chips and functional areas to work collaboratively. This facilitates large-scale matrix multiplication and nonlinear operations, providing strong technical support for end-to-end acceleration of large model inference.

[0108] Figure 7 This is a schematic diagram of the structure of a second computing component provided in an exemplary embodiment of this disclosure.

[0109] In some alternative embodiments, based on any of the above embodiments, such as Figure 7 As shown, the second computing component 222 may include: a nonlinear acceleration subcomponent 2221, configured to: acquire data to be computed, perform nonlinear operator computation tasks based on the data to be computed, and obtain nonlinear operation results.

[0110] The data to be computed can be data from the external host device 20, the controller within the system 20, or memory (e.g., SRAM) to be used for nonlinear operations. For example, data to be used for nonlinear operations such as Softmax, LayerNorm, and GELU activation functions during model inference. The nonlinear acceleration sub-component 2221 is a hardware component for performing nonlinear operator computation tasks, providing hardware acceleration for nonlinear operators in model inference and improving model inference speed. Optionally, the nonlinear acceleration sub-component 2221 may include one or more nonlinear acceleration units to support one or more nonlinear operator computation tasks.

[0111] In this embodiment, the second computing component can perform nonlinear operator computation tasks on the data to be computed through the nonlinear acceleration subcomponent to obtain nonlinear operation results. This enables the second computing component to perform hardware acceleration for nonlinear operators in the model (e.g., a large model), further improving the model inference speed. Furthermore, the matrix multiplication operation implemented by the in-memory computing array of the first computing component helps to achieve end-to-end acceleration of model inference.

[0112] Figure 8 This is a schematic diagram of the structure of a nonlinear accelerator sub-component provided in an exemplary embodiment of this disclosure.

[0113] In some optional embodiments, the nonlinear acceleration sub-component 2221 may include a variety of nonlinear acceleration units 22211 and vector operation units 22212; Any nonlinear acceleration unit 22211 is configured to: acquire the first data to be calculated of the nonlinear acceleration unit 22211, execute the nonlinear operator calculation task corresponding to the nonlinear acceleration unit 22211 based on the first data to be calculated, and obtain the nonlinear operation result corresponding to the nonlinear acceleration unit 22211. The vector operation unit 22212 is configured to acquire second data to be calculated, perform vector operations based on the second data to be calculated, and obtain the vector operation result.

[0114] The various nonlinear acceleration units 22211 can be electrically connected or not connected. These units can be used for hardware acceleration of nonlinear operators in model inference. The configuration of the various nonlinear acceleration units 22211 can be tailored to the nonlinear operators included in the neural network model or large model.

[0115] In some optional embodiments, the various nonlinear acceleration units 22211 may include multiple units such as acceleration units for accelerating the softmax operator, acceleration units for accelerating the layerNorm operator, acceleration units for accelerating the root mean square normalization (RMSNorm) operator, and acceleration units for accelerating one or more activation functions. These activation functions include, but are not limited to, the GELU activation function and the SiLU activation function.

[0116] In some optional embodiments, any one of the various nonlinear acceleration units 22211 can support acceleration of one or more nonlinear operators. For example, nonlinear acceleration unit A can support acceleration of the LayerNorm operator and the RMSNorm operator. Nonlinear acceleration unit B can support acceleration of the GELU activation function and the SiLU activation function.

[0117] The Vector ALU 22212 is a hardware computing unit used to perform vector operations. The Vector ALU can support one or more of the following vector operations: element-wise addition, element-wise multiplication, element-wise division, and others. Element-wise operations refer to the operation on corresponding values ​​of two vectors of the same size. For example, element-wise addition adds the values ​​of corresponding positions of two vectors of the same size. For example, if vector a = [1,2,3] and vector b = [4,5,6], the result of adding a and b element-wise is c = a + b = [1 + 4, 2 + 5, 3 + 6] = [5, 7, 9]. The operations supported by the Vector ALU can be configured according to the computational needs of the neural network model or large model. For example, the Vector ALU can be used to handle residual connections and element-wise operations in Transformers. Optionally, the Vector ALU can be expanded to support more other vector operations as needed.

[0118] In some optional embodiments, the nonlinear acceleration sub-component 2221 may also include other nonlinearly related acceleration units, not limited to the various nonlinear acceleration units 22211 and vector operation units 22212 described above.

[0119] In some optional embodiments, the second computing components 222 at different layers may include the same or different nonlinear acceleration sub-components 2221. In some examples, the second computing components 222 at each layer of the computing chip include the same nonlinear acceleration sub-component 2221, and each nonlinear acceleration sub-component 2221 includes multiple nonlinear acceleration units and vector operation units. In other examples, the second computing components 222 at one layer of the computing chip include multiple nonlinear acceleration units 22211, while the second computing components 222 at another layer of the computing chip include vector operation units 22212. The distribution of components, sub-components, or units in the second functional acceleration processor 22 is not limited.

[0120] In this embodiment, various nonlinear acceleration units and vector operation units are used to support various nonlinear operator operations and vector operations in the model or large model, providing strong technical support for accelerating end-to-end model inference.

[0121] Figure 9 This is a schematic diagram of the structure of a second computing component provided in another exemplary embodiment of this disclosure.

[0122] In some alternative embodiments, such as Figure 9 As shown, the second computing component 222 may further include: an external device input / output unit 2222 and a control unit 2223.

[0123] The external device input / output unit 2222 is configured to receive calculation instructions from the external master device 10; Control unit 2223 is configured as follows: Obtain calculation instructions from external device input / output unit 2222; based on the calculation instructions, determine matrix multiplication calculation tasks and / or nonlinear operator calculation tasks; send the matrix multiplication calculation tasks to the first calculation component 212 of the first functional acceleration processor 21 so that the first calculation component 212 executes the matrix multiplication calculation tasks; and / or control the nonlinear acceleration subcomponent 2221 to execute the nonlinear operator calculation tasks.

[0124] The external device input / output (IO) unit 2222 is a hardware unit used for interaction between the system 20 of this disclosure and external devices (such as the host device 10). The external device input / output unit 2222 may include, but is not limited to, external device input / output and interfaces. Optionally, the external device input / output unit 2222 may include, but is not limited to, a PCIe / CXL high-speed host interface, a chip-to-chip interconnect interface, a DMA (Direct Memory Access) controller, an interrupt controller, a JTAG (Joint Test Action Group) debug interface, and a low-speed peripheral interface. The low-speed peripheral interface may include, but is not limited to, a UART (Universal Asynchronous Receiver / Transmitter), SPI (Serial Peripheral Interface), and I2C (Inter-Integrated Circuit) interface. Among them, the PCIe / CXL high-speed host interface can be used to interact with external hosts (i.e., master devices) for data and instruction exchange; the chip interconnect interface can support the expansion of multiple 3D integrated memory systems, that is, by expanding multiple 3D integrated memory systems, the computing power and bandwidth of the computing hardware can be further improved, thereby further improving the speed of model inference; the DMA controller can be used to manage the transfer of large on-chip data blocks, such as the transfer of data from external FLASH to the memory chip in System 20; the interrupt controller can be used to manage system interrupts and exception events; the JTAG debug interface can support chip debugging and testing in System 20; and the low-speed peripheral interface can be used for configuration, status reading and debugging, etc.

[0125] The computation instructions from the master device 10 are instructions that instruct the system 20 to perform computation tasks. The computation instructions may include, but are not limited to, control fields, operator computation parameters, address information, data format, hardware scheduling control, synchronization status flags, extended reserved bits, etc., to adapt to various operators of the model.

[0126] The control unit 2223 may include, but is not limited to, one or more of a central processing unit (CPU), a coprocessor, and a controller. In some examples, the control unit 2223 may include a lightweight RISC-V (Reduced InstructionSet Computer – Five) or ARM (Advanced RISC Machines) embedded processor core. In other examples, the control unit 2223 may use other embedded processor architectures, such as the ARM Cortex-M / R (ARM Cortex microcontrollers or hard real-time processors) family, Xtensa, Andes, etc., or may use a cluster of multiple small CPU cores.

[0127] The control unit 2223 can be used for operator scheduling and task orchestration during model inference, communication and instruction parsing with external master devices, and coordinated control between the first functional accelerator processor 21 and the second functional accelerator processor 22. In some examples, the calculation instructions issued by the master device are macro instructions. After obtaining the calculation instructions, the control unit 2223 parses the calculation instructions to determine the matrix multiplication calculation task and / or nonlinear operator calculation task.

[0128] Optionally, the control unit 2223 can also be used for exception handling and status monitoring. For example, the master device 10 sends an interrupt request to the control unit 2223. In response to the interrupt request, the control unit 2223 monitors and saves the status of the currently executing computing task to ensure that the computing task can continue to be executed after the interruption. Then, in response to the interrupt request, it executes the exception handling corresponding to the interrupt request. After the exception handling is completed, it resumes the execution of the computing task before the interruption.

[0129] Optionally, the control unit 2223 obtains calculation instructions from the main device 10 through the external device input / output unit 2222, and then determines, based on the calculation instructions, a matrix multiplication calculation task to be executed by the first functional acceleration processor 21 or the first computing component 212, and / or a nonlinear operator calculation task to be executed by the second functional acceleration processor 22 or the second computing component 222. Optionally, the control unit 2223 can parse the calculation instructions according to preset parsing rules to determine the matrix multiplication calculation task and / or the nonlinear operator calculation task. When a matrix multiplication calculation task is determined, the control unit 2223 can send the matrix multiplication calculation task to the first computing component 212 of the first functional acceleration processor 21, so that the first computing component 212 executes the matrix multiplication calculation task. For example, the control unit 2223 can send the matrix multiplication calculation task to the in-memory computing control subcomponent 2121 in the first computing component 212 via a three-dimensional interconnection method, and the in-memory computing control subcomponent 2121 controls the in-memory computing subcomponent 2122 to execute the matrix multiplication calculation task according to the matrix multiplication calculation task. Once a nonlinear operator computation task is determined, the control unit 2223 can schedule the nonlinear acceleration sub-component 2221 in the second computation component 222 to execute the nonlinear operator computation task according to the nonlinear operator computation task.

[0130] In some optional embodiments, the second computing component 222 in each layer of computing chips includes a control unit 2223. Computation instructions from the host device may include computation instructions corresponding to the control units 2223 in one or more layers of computing chips. The control units 2223 in one or more layers distribute matrix multiplication and / or nonlinear operator computation tasks to the components, sub-components, or units that execute the corresponding tasks according to the computation instructions. The computation instructions obtained by control units 2223 in different layers may be different. Optionally, at the same time, the first computing component 212 and the second computing component 222 may collaboratively execute the same computation task or execute different computation tasks respectively. The first computing components 212 in different layers may collaboratively execute the same computation task or execute different computation tasks respectively. The second computing components 222 in different layers may collaboratively execute the same computation task or execute different computation tasks respectively. Different sub-components or different units in the same component may collaboratively execute the same computation task or execute different computation tasks respectively. Specifically, this can be determined according to task scheduling rules, and this disclosure does not limit the specific implementation.

[0131] In some alternative embodiments, the second computing component 222 of each layer includes a control unit 2223.

[0132] In some alternative embodiments, at least one layer of the computing chip may contain only one or a portion of a second computing component 222, which includes a control unit 2223. In this case, the control unit 2223 of that layer or portion of the chip is responsible for operator scheduling and task orchestration during model inference, communication with an external host and instruction parsing, and collaborative control between the first and second functional accelerator processors.

[0133] In this embodiment, the control unit receives calculation instructions from the master device through the external device input / output unit, then parses the calculation instructions and sends the matrix multiplication calculation task to the first computing component of the first functional accelerator, triggering the first computing component to execute the matrix multiplication calculation task. The control unit can also control the nonlinear acceleration sub-component in the second computing component to execute the nonlinear operator calculation task. Thus, the control unit realizes operator scheduling and task orchestration in the model inference process, as well as the collaborative control between the first functional accelerator and the second functional accelerator, enabling the first functional accelerator and the second functional accelerator to collaboratively complete the matrix multiplication operation and nonlinear operator operation in the model inference process, which helps to realize end-to-end hardware acceleration of model inference.

[0134] In some alternative embodiments, such as Figure 9 As shown, the second computing component 222 may further include: a second three-dimensional interconnect interface 2224; The control unit 2223 is specifically configured to send the matrix multiplication calculation task to the first computing component 221 via the second three-dimensional interconnect interface 2224.

[0135] The second three-dimensional interconnect interface 2224 is a data and control channel between the second computing component 222 and the chip of the adjacent layer, and can be used for high-speed data transmission between the second computing component 222 and other layers.

[0136] Optionally, the second three-dimensional interconnect interface 2224 may be a three-dimensional TSV interface. The second three-dimensional interconnect interface 2224 may include, but is not limited to, a transmitter, a receiver, and bonding interface logic. In some examples, the second three-dimensional interconnect interface 2224 may include a TSV transmitter, a TSV receiver, and hybrid bonding interface logic. The transmitter can be used to send matrix multiplication calculation tasks via TSV to the first computing component 212 or other layers. The receiver can be used to receive data to be calculated from other layers. The hybrid bonding interface logic can be used to manage the physical layer communication protocol with the hybrid bonding pad.

[0137] In this embodiment, high-speed data transmission between layers is achieved through a three-dimensional interconnect interface, enabling multi-layer chips and functional areas to work collaboratively. This facilitates large-scale matrix multiplication and nonlinear operations, providing strong technical support for end-to-end acceleration of large model inference.

[0138] In some optional embodiments, the external device input / output unit 2222 is also configured to receive weight data from the master device 10; The control unit 2223 is also configured to write weight data from the external device input / output unit 2222 into the first storage component 211.

[0139] The weight data from the master device 10 can be the weight data of the model to be accelerated inference.

[0140] In some optional embodiments, the master device 10 can read weight data from the external memory of the system 20, generate a write instruction for the weight data, and send the write instruction to the second computing component 222. The control unit 2223 in the second computing component 222 obtains the write instruction through the external device input / output unit 2222 and writes the weight data into the first storage component 211 according to the write instruction. Optionally, the write instruction may include the destination address of the weight data in the first storage component 211, and the control unit 2223 can write the weight data into the first storage component 211 according to the destination address in the write instruction.

[0141] In some optional embodiments, the master device 10 can send a data transfer instruction to the second computing component 222. The data transfer instruction may include the source address and destination address of the weight data. The source address may be the storage address of the weight data in a memory (e.g., FLASH) outside the system 20. The control unit 2223 can obtain the data transfer instruction through the external device input / output unit 2222, read the weight data from the storage space corresponding to the source address according to the data transfer instruction, and write the weight data into the first storage component 211. Optionally, the control unit 2223 triggers the DMA controller to perform a data transfer operation. For example, the DMA controller reads the weight data from the storage space corresponding to the source address and writes the weight data into the first storage component 211.

[0142] In this embodiment, before executing the model inference task, the weight data is transferred from external memory to the first storage component within the system in one go. The large capacity of the first storage component stores the model's weight data, avoiding remote data transfer between the external and internal memory during inference. This effectively reduces inference latency caused by data transfer and further improves model inference speed. Furthermore, by pre-writing the weight data to the first storage component, which is integrated with the first computing component in a three-dimensional stack, the data transmission path is effectively shortened, further reducing transmission latency and data transfer power consumption.

[0143] In some optional embodiments, the various nonlinear acceleration units 22211 may include a variety of the following acceleration units: normalized exponential acceleration unit, layer normalization acceleration unit, root mean square normalization acceleration unit, and activation function acceleration unit.

[0144] The normalized exponent acceleration unit is configured to perform normalized exponent operations on the data to be computed input to the acceleration unit.

[0145] The layer normalization acceleration unit is configured to perform layer normalization operations on the data to be computed input to the acceleration unit.

[0146] The root mean square normalization acceleration unit is configured to perform root mean square normalization on the data to be computed input to the acceleration unit.

[0147] The activation function acceleration unit is configured to perform a preset activation function operation on the data to be computed input to the acceleration unit.

[0148] Among them, the normalized exponent acceleration unit can be called the Softmax acceleration engine, which is used to accelerate the Softmax operator.

[0149] The layer normalization acceleration unit, also known as the LayerNorm acceleration engine, is used to accelerate the LayerNorm operator.

[0150] The root mean square normalization acceleration unit, also known as the RMSNorm acceleration engine, is used to accelerate the RMSNorm operator.

[0151] The activation function acceleration unit is used to accelerate one or more activation functions.

[0152] In some optional embodiments, acceleration of layer normalization operators and root mean square normalization operators can be supported through a normalization acceleration unit. This normalization acceleration unit, also known as a normalization engine, is used to accelerate the LayerNorm and RMSNorm operators.

[0153] In some optional embodiments, the Softmax acceleration engine may include, but is not limited to, dedicated hardware circuits such as an Exponential Unit, a Max-Finding Unit, a Reciprocal / Division Unit, and an Accumulation and Reduction Unit. These hardware circuits enable full pipelined hardware acceleration of the Softmax operator in the Attention mechanism. Optionally, the Softmax acceleration engine may employ piecewise linear approximation or a lookup table (LUT) combined with low-precision interpolation to achieve efficient approximate calculation of the exponential function.

[0154] In some optional embodiments, the LayerNorm acceleration engine, RMSNorm acceleration engine, or normalization engine may include, but is not limited to, hardware circuits such as mean / variance calculation units, square root reciprocal calculation units, and element-wise multiply-add units, which support hardware acceleration of normalization operators such as LayerNorm and RMSNorm in the model or large model. The Activation Engine can perform efficient hardware computation of nonlinear activation functions such as GELU and SiLU through piecewise polynomial approximation or LUT-based approximation schemes.

[0155] In some alternative embodiments, the nonlinear acceleration sub-component 2221 can be implemented using programmable FPGA (Field Programmable Gate Array) logic to provide greater algorithmic flexibility.

[0156] In some alternative embodiments, the acceleration of nonlinear operators such as Softmax and LayerNorm can also be achieved using hardware implementations based on the CORDIC (Coordinate Rotation Digital Computer) algorithm.

[0157] In this embodiment, various nonlinear operators in the model or large model are accelerated through nonlinear acceleration units such as normalized exponential acceleration units, layer normalization acceleration units, root mean square normalization acceleration units, and activation function acceleration units, providing strong technical support for achieving end-to-end acceleration of model inference.

[0158] In some optional embodiments, the first computing component 212 may include: a memory computing control subcomponent 2121 and a memory computing integrated computing subcomponent 2122; the second computing component 222 may include: a variety of nonlinear acceleration units 22211, a vector operation unit 22212, a control unit 2223 and an external device input / output unit 2222.

[0159] The external device input / output unit 2222 is configured to receive weight data of a preset model from the external master device 10 after the system 20 is powered on.

[0160] The control unit 2223 is configured to write weight data to the first storage component 211.

[0161] The in-memory computing control subcomponent 2121 is configured to: read weight data from the first storage component 211 and write the weight data into the in-memory computing subcomponent 2122 in response to the weight data loading instruction of the control unit 2223.

[0162] The control unit 2223 is also configured to: acquire calculation instructions from the preset model of the master device 10, and based on the calculation instructions, control the in-memory computing control sub-component 2121, the in-memory computing sub-component 2122, the multiple nonlinear acceleration units 22211 and the vector operation unit 22212 to work together to execute the inference calculation task of the preset model.

[0163] In this system, system 20 is powered on, providing power to enable it to start working and perform computational tasks that accelerate model inference. The structure and function of the first computing component 212 and the second computing component 222 can be referenced in the aforementioned embodiments. The preset model can include any neural network model. Optionally, the preset model may include, but is not limited to, deep learning models such as Transformer-type models, CNN (Convolutional Neural Network) models, RNN (Recurrent Neural Network) models, LSTM (Long Short-Term Memory) models, Mamba (or state-space model) models, and MoE (Mixture of Experts) models. Acceleration of different neural network models can be adapted by adjusting the layout of the weight data in the in-memory array of the first computing component 212 and the type configuration of the activation function in the second computing component 222.

[0164] The weight data loading instruction is used to instruct the in-memory computing control subcomponent 2121 to load weight data into the in-memory computing subcomponent 2122. The weight data loading instruction may include the source address of the weight data in the first storage component 211, the destination addressing information of the weight data within the in-memory computing array, the data format and quantization configuration information of the weight data, control signals, etc. The in-memory computing control subcomponent 2121 can read the weight data from the first storage component 211 according to the source address in the weight data loading instruction, and write the weight data into the storage cells of the in-memory computing subcomponent 2122 according to the destination addressing information in the weight data loading instruction. The destination addressing information is used to specify the specific location where the weight will be written into the array. The destination addressing information may include, but is not limited to, the row address (word line address), column address (bit line select address), subarray bank (memory bank), offset address, etc., of the storage cell. The addressing principle of the in-memory computing array for any layer of memory chip is prior art and will not be elaborated upon in this disclosure.

[0165] After loading the weight data into the storage unit of the in-memory computing array in the in-memory computing sub-component 2122, the master device 10 can send calculation instructions for the preset model to the control unit 2223. The master device 10 parses the calculation instructions to generate matrix multiplication calculation tasks, nonlinear operator calculation tasks, etc., and controls the sub-components in the first computing component 212 and the units in the sub-components of the second computing component 222 to work collaboratively to complete the end-to-end inference task of the preset model. Optionally, the working sequence of each sub-component and unit can be determined based on the data dependencies between various operators of the preset model, and the work of each sub-component and unit can be scheduled according to the timing to accelerate the inference of the preset model. For example, the matrix multiplication operation in the Attention mechanism and FFN can be calculated by a sub-component in the first computing component 212, such as calculating the query vector Q, key vector K, and value vector V. Then, the vector operation unit in the second computing component 222 performs the matrix multiplication of the transpose of Q and K, or the first computing component 212 performs the matrix multiplication of the transpose of Q and K to obtain the attention weight (or attention score). Then, the Softmax acceleration engine performs Softmax normalization on the Attention Score to obtain the normalized result. The normalized result is then weighted and summed with V to obtain the attention operation result (i.e., the Attention output). The LayerNorm acceleration engine then normalizes the Attention output to complete the calculation of the entire Attention sub-layer and obtain the output of the Attention sub-layer. The output of the Attention sub-layer is transmitted to the first functional accelerator 21 via a 3D interconnect interface. The first computing component 212 in the first functional accelerator 21 performs matrix multiplication operations on the fully connected layers of the FFN in computation mode, and sends the intermediate results to the second functional accelerator 22. The activation function acceleration engine in the second functional accelerator 22 accelerates the computation of nonlinear activation functions (e.g., GELU, SiLU activation functions). Then, the vector operation unit in the second functional accelerator 22 performs element-wise addition operations on the residual connections, and the normalization acceleration engine performs normalization processing. At this point, the inference of a complete Transformer Block is completed. For multi-layer Transformer Blocks, the above process can be repeated until the computation of all layers is completed, obtaining the inference result of the Transformer model. The inference result is then returned to the external host via the PCIe / CXL interface.Through the above workflow, matrix multiplication operations in large model inference can be performed through the in-memory array in the first functional accelerator 21, without needing to read weight data from the in-memory array. Nonlinear operations are always completed in the second functional accelerator 22 through a dedicated hardware acceleration engine. Only necessary intermediate activation results need to be transferred between the two functional areas, and the amount of data is much smaller than that of the weight data, thereby minimizing data transfer overhead and achieving end-to-end high-energy-efficiency inference acceleration of the model. It should be noted that this explanation of end-to-end inference acceleration only uses the Transformer model as an example. In practical applications, the system 20 disclosed herein is not limited to supporting inference acceleration of the Transformer model.

[0166] In this embodiment, the weight data of the preset model is transferred from external memory to internal storage component through hardware such as sub-components and units in the first and second computing components. The weight data of the preset model can be pre-written into the first storage component before inference. During inference, this avoids or reduces the number of data transfers between external and internal memory. Furthermore, within the system, the storage chip and computing chip are stacked and integrated, effectively increasing data bandwidth. The transmission distance between the internal memory and the computing component is greatly shortened, effectively reducing data latency and improving inference efficiency. In addition, the first computing component supports matrix multiplication operations of various scales, and the second computing component supports operations with various nonlinear operators. The collaborative work of the first and second computing components enables end-to-end inference acceleration of the model, further improving model inference efficiency.

[0167] In some optional embodiments, the system 20 of this disclosure may include a main controller for operator scheduling and task orchestration during model inference, communication with an external host and instruction parsing, and collaborative control between the first functional accelerator processor and the second functional accelerator processor. Optionally, the main controller may be an independent component separate from the first functional accelerator processor 21 and the second functional accelerator processor 22. The controller may be located on any layer of computing chips in at least one layer of computing chips.

[0168] In some alternative embodiments, the main controller may be located in the first computing component. That is, the first computing component includes a main controller for operator scheduling and task orchestration during model inference, communication with an external host and instruction parsing, and collaborative control between the first functional accelerator and the second functional accelerator. The control unit of the second computing component may not have this function. The control unit may be used to control the collaborative work of sub-components or units within the second computing component.

[0169] In some alternative embodiments, Figure 10This is a schematic diagram of the structure of a three-dimensional integrated in-memory computing system provided in another exemplary embodiment of this disclosure. For example... Figure 10 As shown, the three-dimensional integrated in-memory computing system 20 of this embodiment may include multilayer chips and a packaging substrate. The multilayer chips may include multilayer memory chips (i.e., memory dies in the figure, also referred to as memory dies or memory chips) and multilayer computing chips (i.e., computing dies in the figure, also referred to as computing dies or computing chips). The first functional area represents the area corresponding to the first functional accelerator 21, and the second functional area represents the area corresponding to the second functional accelerator 22. The first functional accelerator 21 may be represented as a compute matrix processor (CIMMP), and the second functional accelerator 22 may be represented as a nonlinear neuromorphic processor (NNP). The multilayer chips are three-dimensionally integrated and interconnected in the vertical direction using TSVs and hybrid bonding methods. The hybrid bonding method in the figure is exemplified by copper-copper (Cu-Cu) hybrid bonding. That is, the RDL layer and the TSV are bonded through Cu-Cu hybrid bonding. The RDL layer of the chip close to the packaging substrate is connected to the packaging substrate through a micro-bump array. The packaging substrate is connected to the PCB (Printed Circuit Board) via a solder bump array to interconnect with the host device, other devices, or components via the PCB.

[0170] In some alternative embodiments, the physical partitioning of the first functional accelerator processor and the second functional accelerator processor may not be based on storage dies and compute dies, but rather all storage dies may be stacked uniformly at the bottom layer and all compute dies may be stacked uniformly at the top layer.

[0171] In some alternative embodiments, the first functional accelerator processor (CIMMP) and the second functional accelerator processor (NNP) can be horizontally tiled on the same layer of die, and then multiple layers of dies can be vertically stacked to achieve multiple CIMMPs and multiple NNPs; In some alternative embodiments, CIMMP and NNP can be integrated on the same die without distinguishing between different die layers, which is suitable for inference acceleration of small-scale models.

[0172] In some alternative embodiments, multi-layer dies can be interconnected via a TSV+Hybrid Bonding interconnection network, such as... Figure 10 As shown, this can be represented by columnar structures that vertically penetrate each die layer and interlayer bonding interfaces. TSV silicon vias penetrate the silicon substrate of each die layer, and high-density connections are achieved between adjacent dies through hybrid bonding interfaces (Cu-Cu pads + SiO2-SiO2 dielectric bonding).

[0173] In some optional embodiments, the thickness of each die layer after thinning can be, but is not limited to, 20-50 μm, and the spacing between dies (Hybrid Bonding interface) can be, but is not limited to, 1-10 μm.

[0174] In some alternative embodiments, the diameter of the TSV can be, but is not limited to, 1-5 μm, and the aspect ratio can be, but is not limited to, 10:1 to 20:1. The interior of the TSV can be filled with conductive materials such as copper (Cu) or tungsten (W), and the exterior can be wrapped with a silicon dioxide (SiO2) insulating layer to prevent leakage and crosstalk.

[0175] In some optional embodiments, ultra-high density interlayer interconnects are achieved on the mating surfaces of adjacent dies through a hybrid bonding method combining Cu-Cu direct bonding and SiO2-SiO2 direct bonding. The typical pad pitch of hybrid bonding can be, but is not limited to, 1-10 μm, far smaller than traditional micro-bump solutions (approximately 40-55 μm pitch), achieving an interconnect density increase of 1-2 orders of magnitude. The data bus between the storage die and the compute die is interconnected via TSV+Hybrid Bonding, achieving an interconnect density of up to 102 per square millimeter. 4 -10 6 There are several connection points. Data exchange between different functional areas (CIMMP and NNP) can be achieved through a shared three-dimensional interconnect bus (including inter-layer and intra-layer buses); critical data paths (such as the transmission of matrix multiplication results from CIMMP to NNP, and the transmission of nonlinear calculation results from NNP to CIMMP) can be allocated dedicated high-bandwidth TSV channels; control signals and configuration signals can be transmitted through independent low-bandwidth TSV channels.

[0176] Figure 11 This is a schematic diagram of the structure of a compute matrix processor (CIMMP) provided in an exemplary embodiment of this disclosure. Figure 11As shown, the in-memory compute matrix processor (CIMMP) may include multiple sub-processor cores (CIMMP sub-cores), for example, each compute die layer includes one CIMMP sub-core, which may correspond to the first compute component described above. Each CIMMP sub-core may include, but is not limited to, an in-memory controller (not shown in the figure), an input data converter and a buffer, an in-memory compute array, an adder tree (i.e., an adder tree), an accumulator, TSV pillars, and an interface. The in-memory controller may refer to the in-memory control sub-component 2121 described above. The input data converter can be used to convert the data type or format of the input data. The buffer may refer to a cache used to cache input data or converted input data. The in-memory compute array may include in-memory compute units (i.e., in-memory compute units) that support read / write and parallel computing. The adder tree is used to perform addition operations on the multiplication results of each column in the in-memory compute array. The accumulator is the accumulation unit 21224 described above, used to accumulate partial sums and results across cycles. The TSV pillars and the interface represent the first three-dimensional interconnect interface described above. The specific functions of each part in the CIMMP sub-Core can be found in the aforementioned embodiments, and will not be repeated here.

[0177] In some optional examples, the in-memory computing unit may include a 6T SRAM memory cell (containing six transistors: M1-M6, forming a cross-coupled inverter pair and two transmission gates) and a 4T NOR gate (containing four transistors). One input of the NOR gate is connected to the QB terminal of the SRAM cell (storing the inverted value of the weight data), and the other input is connected to the input line driver (receiving the inverted value INB of the input activation data). The output Y of the NOR gate represents the single-bit multiplication result. Each column of the array corresponds to an adder tree, summing the outputs of multiple in-memory computing units in the same column. The input line driver is responsible for converting multi-bit input data into a time-division multiplexed single-bit input signal sequence, driving it to the NOR gate input (INB) of each row of in-memory computing units. For 8-bit input data, the input line driver requires 8 computation cycles to sequentially send out 8 single-bit signals from LSB to MSB. It should be noted that this is only an exemplary implementation of the in-memory computing unit, and actual applications are not limited to this example.

[0178] In some optional embodiments, the CIMMP subcore may also include a multibit input transfer logic for receiving multi-cycle portions and results, shifting and accumulating them according to bit weights, and converting multi-cycle single-bit multiply-accumulate results into multi-bit × multi-bit multiply-accumulate results.

[0179] Figure 12 This is a schematic diagram of the structure of a nonlinear neuromorphic processor (NNP) provided in an exemplary embodiment of this disclosure. Figure 12 As shown, a nonlinear neuromorphic processor (NNP) may include a nonlinear vector processor (NLVP), a vector processing unit, a control CPU, an on-chip SRAM cache, conventional peripheral I / O interfaces, TSV pillars, and interfaces. The nonlinear vector processor (NLVP) includes various nonlinear acceleration units 22211. These nonlinear acceleration units 22211 include activation function acceleration engines such as Softmax, LayerNorm / RMSNorm, and GELU / SiLU. The vector processing unit (VPU) may refer to the aforementioned vector operation unit 22212. The vector processing unit supports vector operations on data in various formats (e.g., INT4, INT8, INT16, FP8, BF16, FP16, etc.). The control CPU refers to the aforementioned control unit 2223. The on-chip SRAM cache can be used to store intermediate activation results, lookup table data, etc. The conventional peripheral I / O interfaces may refer to the aforementioned external device input / output units 2222. The TSV copper pillars and interfaces can refer to the aforementioned second three-dimensional interconnect interface 2224. The specific structure and function of each component in the nonlinear neuromorphic processor (NNP) can be found in the foregoing embodiments, and will not be elaborated upon here.

[0180] In some optional embodiments, the vector operation unit 22212 may include one or more of a multi-channel parallel element-wise adder array, an element-wise multiplier array, and an element-wise divider array.

[0181] In some alternative embodiments, the control CPU may include an instruction cache (I-Cache) unit, a data cache (D-Cache) unit, an instruction decoding unit, an execution unit, and a system bus. The on-chip SRAM cache may be a local cache tightly coupled to each acceleration engine of NLVP, used to temporarily store input data, intermediate calculation results, lookup table data, etc.

[0182] Figure 13 This is a flowchart of the end-to-end inference workflow of a model provided in an exemplary embodiment of this disclosure.

[0183] In some optional instances, taking the Transformer model as an example, the end-to-end inference workflow of the model may include the following steps: Start node: System 20 powers on and receives inference requests.

[0184] Step S1 (Weight Preloading): Receive weights via PCIe / CXL → Configure Memory Mode on the in-memory controller → Write weight data to the in-memory array → Determine if all weight data has been loaded. If yes, proceed to S2; otherwise, continue loading.

[0185] Step S2 (Input Data Reception): Receive input Token (word or token) → Token Embedding → Input data is sent to CIMMP input line driver.

[0186] Step S3 (Q / K / V Projection - Matrix Multiplication): CIMMP enters CIM Mode (computation mode) → Q weight matrix multiplication → K weight matrix multiplication → V weight matrix multiplication → The result is sent to NNP via TSV.

[0187] Step S4 (Attention Score Calculation - Non-linear Operation): NNP receives Q and K → calculates Q×K (Optional CIMMP acceleration) → Softmax normalization → Weighted summation with V → LayerNorm normalization.

[0188] Step S5 (FFN - Matrix Multiplication + Nonlinearity): Attention output is sent to CIMMP via TSV → CIMMP performs Input matrix multiplication → Intermediate results are sent to NNP via TSV → GELU / SiLU activation function calculation → CIMMP performs Hidden matrix multiplication → Result is sent to NNP via TSV.

[0189] Step S6 (Residual Join + Normalization): Residual Join (element-wise addition) → LayerNorm / RMSNorm Normalization.

[0190] Step S7 (Iterative Judgment): Have all Transformer Blocks been completed? (No → Return to S3 to process the next layer, Yes → Continue).

[0191] Output steps: Final Logits output → returned to the host via PCIe / CXL → Token sampling / decoding.

[0192] End point: Reasoning complete.

[0193] The steps performed in CIMMP include the matrix multiplication parts in S3 and S5, while the steps performed in NNP include the nonlinear parts in S4 and S5 and S6.

[0194] In 2D chip architectures of related technologies, computing units such as GPUs (Graphics Processing Units), NPUs (Neural Processing Units), and TPUs (Tensor Processing Units) exchange data with storage units (such as HBM, DRAM, and SRAM) through interposers or PCB-level interconnects. This architecture suffers from "memory walls" and "power walls," with data exchange between the processor and memory occurring through limited bus bandwidth. During large model inference, the generation of each token requires repeatedly reading massive amounts of weight data from memory. For example, in the LLaMA-70B model, the weight data alone exceeds 140GB. Each inference iteration requires moving this weight data from memory to the computing unit, leading to severe bandwidth bottlenecks and energy efficiency losses. Statistics show that in large model inference based on traditional architectures, the energy consumed by data movement can account for 60%-80% of the total system energy consumption. Furthermore, 2D chips, limited by their planar area and heat dissipation capabilities, struggle to achieve the computational density required for large model inference within a limited power budget. Furthermore, when performing matrix multiplication, the weight data must first be read from the memory, transmitted to the computing unit via the bus, and then the intermediate results must be written back to the memory after the calculation is completed. This results in a huge data transfer overhead, and the limited number of memory read interfaces restricts the system throughput.

[0195] To address the aforementioned problems in related technologies, the three-dimensional integrated in-memory computing system 20 of this disclosure achieves heterogeneous three-dimensional integration of a first functional acceleration processor for matrix multiplication operations and a second functional acceleration processor for nonlinear operations, which has the following beneficial effects: (1) Ultra-high interconnect bandwidth: Through TSV+Hybrid Bonding hybrid bonding 3D integration technology, the interconnect spacing between the storage die and the computing die is reduced from 40-55μm in the traditional micro-bump to 1-10μm, increasing the interconnect density by more than an order of magnitude. Compared with the traditional 2D chip architecture, this achieves an interconnect bandwidth improvement of approximately 5.13 times. This means that during large model inference, the transmission bandwidth of weight data and intermediate activation results will no longer constitute a system bottleneck.

[0196] (2) Ultra-high energy efficiency: Through the in-memory computing array in the in-memory computing matrix processor, the weight data always resides in the in-memory computing unit during the inference process, and the matrix multiplication operation is completed directly on the storage side, which greatly reduces the number of times the weight data is repeatedly transported. Combined with the low-power three-dimensional interconnect of TSV+Hybrid Bonding, the overall energy efficiency of the system can be improved by more than 5 times compared with the traditional 2D chip architecture. Specifically, the energy efficiency improvement comes from three aspects: First, in-memory computing eliminates the power consumption of transporting weight data (accounting for about 60%-80% of the total power consumption of the traditional architecture); second, TSV+Hybrid Bonding shortens the signal transmission distance from the millimeter level to the micrometer level, which greatly reduces the power consumption of data transmission; third, the nonlinear neuromorphic processor processes nonlinear operations through a dedicated hardware acceleration engine, which can improve the energy efficiency of nonlinear operators by 1-2 orders of magnitude compared with general processors.

[0197] (3) End-to-end large model inference acceleration: Through the heterogeneous fusion design of in-memory matrix processor and nonlinear neuromorphic processor, the system 20 disclosed herein can fully cover all operator types (matrix multiplication and nonlinear operators) in large model inference, realizing true end-to-end in-memory acceleration. Compared with the design of related technical solutions that only accelerate some operators, it avoids frequent data format conversion and inter-processor communication overhead, and significantly reduces end-to-end inference latency.

[0198] (4) High computing resource utilization: The two functional areas have been specially designed with hardware optimization for different operator types—the in-memory matrix processor performs high-parallel in-memory acceleration for regular matrix multiplication with large data volume; the nonlinear neuromorphic processor performs flexible pipeline acceleration for irregular nonlinear operators with diverse computation modes. This heterogeneous division of labor design avoids the waste of resources in homogeneous computing architecture and greatly improves the utilization of computing resources.

[0199] (5) Good scalability: This disclosure adopts a modular design of stacked dies, and the number of M storage dies and N computation dies can be flexibly configured according to the inference needs of large models of different sizes. For larger models (such as LLaMA-405B), more weight parameters can be accommodated by increasing the number of storage dies; for scenarios requiring higher throughput, the parallel computing capability can be improved by increasing the number of computation dies. In addition, the size of the in-memory computing array can also be flexibly adjusted, for example, 512×512, 1024×1024 or larger, to adapt to the scale of layered parameters of different models.

[0200] (6) The "read interference write" problem is fundamentally solved: The in-memory matrix processor in this disclosure adopts a digital domain in-memory computing solution, and the CIM Mode and Memory Mode are completely independently controlled. In CIM Mode, the NOR gate output in the in-memory computing unit directly drives the addition tree, which will not affect the bit line voltage of the storage unit, thus eliminating the "read interference write" problem in the traditional analog domain CIM solution in principle.

[0201] The various embodiments and optional examples disclosed herein can be implemented individually or in any combination without conflict, and can be set according to actual needs.

[0202] Exemplary methods Figure 14 This is a flowchart illustrating a model inference method based on a three-dimensional integrated in-memory computing system provided in an exemplary embodiment of this disclosure. The method provided in this embodiment can be implemented using any of the three-dimensional integrated in-memory computing systems provided in the above embodiments. Figure 14 As shown, the method includes performing the following steps through the three-dimensional integrated in-memory computing system provided in any of the above embodiments: Step 510: Obtain the weight data of the preset model and write the weight data of the preset model into the in-memory array in the first computing component.

[0203] Step 520: Obtain the input data for the preset model.

[0204] Step 530: Based on the input data, control the first computing component and the second computing component to work together to perform inference calculation tasks on the preset model.

[0205] During the inference process, the matrix multiplication calculation task in the preset model is executed by the in-memory computing array in the first computing component, and the nonlinear operator calculation task in the preset model is executed by the nonlinear acceleration subcomponent in the second computing component.

[0206] In some optional embodiments, step 530, which controls the first and second computing components to work together based on the input data to perform inference calculations on a preset model, may include: The input data is driven to the in-memory computing array via the in-memory control subcomponent in the first computing component; matrix multiplication is performed by the in-memory computing array to obtain query vectors, key vectors, and value vectors; matrix multiplication is performed by the vector operation unit in the second computing component based on the query vectors and key vectors to obtain attention weights; the nonlinear acceleration subcomponent in the second computing component determines the attention operation result based on the attention weights and value vectors, and the output result of the current attention layer of the preset model is determined based on the attention operation result. In response to the current attention layer not being the last attention layer, the output result of the current attention layer is used as input data, and the following steps are iteratively executed: the input data is driven to the in-memory computing array via the in-memory control subcomponent in the first computing component; matrix multiplication is performed by the in-memory computing array to obtain query vectors, key vectors, and value vectors; matrix multiplication is performed by the vector operation unit in the second computing component based on the query vectors and key vectors to obtain attention weights; the nonlinear acceleration subcomponent in the second computing component determines the attention operation result based on the attention weights and value vectors, and the output result of the current attention layer of the preset model is determined based on the attention operation result. In response to the current attention layer being the last attention layer, the output of the current attention layer is determined as the inference result of the preset model.

[0207] Here, the attention layer can refer to the Transformer Block, and the current attention layer refers to the Transformer Block for the current inference.

[0208] The specific operations and hardware for performing each step in the embodiments of this disclosure can be found in the foregoing embodiments, for example, but not limited to, the following: Figure 13 The embodiments shown are not described in detail here.

[0209] This disclosure also provides an electronic device, including the three-dimensional integrated memory system provided in any of the above embodiments.

[0210] In some alternative embodiments, the electronic device may include one or more three-dimensional integrated in-memory computing systems.

[0211] In some optional embodiments, the electronic device may further include: a host device 10 configured to: run an application program of a preset model, generate computational tasks for the 3D integrated in-memory computing system, invoke the 3D integrated in-memory computing system to execute the computational tasks, and complete the inference of the preset model. (See reference...) Figure 1 As shown.

[0212] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0213] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0214] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0215] The system or apparatus of this disclosure may be implemented in many ways. For example, it may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps of the method in a circuit is for illustrative purposes only, and the steps of the method of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the method according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the method according to this disclosure.

[0216] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0217] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0218] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A three-dimensional integrated in-memory computing system, comprising: A multi-layer chip, the multi-layer chip comprising at least one layer of memory chip and at least one layer of computing chip; The multi-layer chip is integrated using a three-dimensional stacking method in the vertical direction; The components in the multi-layer chip include a first functional acceleration processor and a second functional acceleration processor; The first functional acceleration processor includes a first storage component in the storage components of the at least one layer of storage chip, and a first computing component in the computing components of the at least one layer of computing chip; The first storage component is configured to store the data required by the first computing component to perform matrix multiplication calculation tasks; The first computing component is configured to perform matrix multiplication calculation tasks via the in-memory array in the first computing component; The second functional acceleration processor includes a second storage component in the storage components of the at least one layer of storage chip, and a second computing component in the computing components of the at least one layer of computing chip; The second storage component is configured to store the data required by the second computing component to perform nonlinear operator computation tasks; The second computing component is configured to perform nonlinear operator computation tasks through a nonlinear acceleration subcomponent in the second computing component.

2. The system according to claim 1, wherein, The first computing component includes: a memory computing control subcomponent and a memory computing integrated computing subcomponent; The memory computing control subcomponent is configured as follows: The working mode of the in-memory computing subcomponent is controlled to be storage mode. According to the weight data loading instruction, the weight data participating in the matrix multiplication calculation task is read from the first storage component and the weight data is written into the in-memory computing subcomponent. The working mode of the in-memory computing sub-component is updated from storage mode to computing mode. It obtains a matrix multiplication calculation task from the main device or the second computing component, and controls the in-memory computing sub-component to execute the matrix multiplication calculation task to obtain the matrix multiplication operation result.

3. The system according to claim 2, wherein, The in-memory computing sub-component includes: an input driving unit, an in-memory computing array, an addition unit, and an accumulation unit; The in-memory computing control subcomponent is specifically configured to: determine the first input data of the in-memory computing subcomponent according to the matrix multiplication calculation task, and transmit the first input data to the input driving unit of the in-memory computing subcomponent; The input driving unit drives the first input data to the in-memory computing array; The in-memory computing array performs a multiplication operation based on the data from the input driving unit and the weight data pre-written in the in-memory computing array to obtain the multiplication result. The addition unit performs an addition operation on the result of the multiplication operation to obtain the current part and the result; In response to the fact that the current partial sum and result is not the partial sum and result of the last cycle, the accumulation unit performs an addition operation on the current partial sum and result and the previous accumulation result to determine the current accumulation result; or, in response to the fact that the current partial sum and result is the partial sum and result of the last cycle, the current accumulation result is used as the result of the matrix multiplication operation; the previous accumulation result is the accumulation result obtained by the accumulation unit in the previous addition operation.

4. The system according to claim 2, wherein, The in-memory control sub-component includes: The mode control register is configured to store the operating mode of the in-memory computing sub-component; The timing control unit is configured to: read the working mode from the mode control register, and control the working timing of the in-memory computing sub-component according to the working mode; The data stream scheduling unit is configured to determine the address signal based on the working mode and the working timing. The address decoding unit is configured to convert the address signal into a word line selection signal in the in-memory computing subcomponent according to the operating mode and the operating timing.

5. The system according to claim 2, wherein, The first computing component further includes: The first three-dimensional interconnect interface is used for vertical data transmission between multi-layer chips.

6. The system according to claim 1, wherein, The second computing component includes: The nonlinear acceleration subcomponent is configured to: acquire data to be calculated, perform nonlinear operator calculation tasks based on the data to be calculated, and obtain nonlinear operation results.

7. The system according to claim 6, wherein, The nonlinear acceleration sub-component includes various nonlinear acceleration units and vector operation units; Any of the nonlinear acceleration units is configured to: acquire first data to be computed by the nonlinear acceleration unit, execute a nonlinear operator computation task corresponding to the nonlinear acceleration unit based on the first data to be computed, and obtain a nonlinear operation result corresponding to the nonlinear acceleration unit. The vector operation unit is configured to acquire second data to be calculated, perform vector operations based on the second data to be calculated, and obtain the vector operation result.

8. The system according to claim 6, wherein, The second computing component also includes: The external device input / output unit is configured to receive computing instructions from an external master device; The control unit is configured as follows: Obtain the calculation instructions from the external device input / output unit; Based on the computation instructions, determine the matrix multiplication computation task and / or the nonlinear operator computation task; The matrix multiplication calculation task is sent to the first computing component of the first functional acceleration processor, so that the first computing component executes the matrix multiplication calculation task; and / or, the nonlinear acceleration subcomponent is controlled to execute the nonlinear operator calculation task.

9. The system according to claim 8, wherein, The second computing component further includes: a second three-dimensional interconnect interface; The control unit is specifically configured to send the matrix multiplication calculation task to the first computing component via the second three-dimensional interconnect interface.

10. The system according to claim 8, wherein, The external device input / output unit is further configured to receive weight data from the master device; The control unit is also configured to write the weight data from the external device input / output unit into the first storage component.

11. The system according to any one of claims 1-10, wherein, The first computing component includes: a memory computing control subcomponent and a memory computing integrated computing subcomponent; The second computing component includes: various nonlinear acceleration units, vector operation units, control units, and external device input / output units; The external device input / output unit is configured to receive weight data from a preset model of an external master device after the system is powered on. The control unit is configured to write the weight data into the first storage component; The in-memory computing control subcomponent is configured to: in response to the weight data loading instruction of the control unit, read the weight data from the first storage component and write the weight data into the in-memory computing subcomponent; The control unit is further configured to: acquire calculation instructions from the preset model of the main device, and based on the calculation instructions, control the in-memory computing control subcomponent, the in-memory computing integrated subcomponent, the various nonlinear acceleration units and the vector operation unit to work together to execute the inference calculation task of the preset model.

12. The system according to claim 11, wherein, The various nonlinear acceleration units include several of the following acceleration units: The normalized exponent acceleration unit is configured to perform normalized exponent operations on the data to be computed input to the acceleration unit. The layer normalization acceleration unit is configured to perform layer normalization operations on the data to be computed input to the acceleration unit; The root mean square normalization acceleration unit is configured to perform root mean square normalization on the data to be computed input to the acceleration unit. The activation function acceleration unit is configured to perform a preset activation function operation on the data to be computed input to the acceleration unit.

13. A model reasoning method based on a three-dimensional integrated in-memory computing system, comprising: The following reasoning process is performed using the three-dimensional integrated in-memory computing system as described in any one of claims 1-12: Obtain the weight data of the preset model and write the weight data of the preset model into the in-memory computing array in the first computing component; The system acquires input data from the preset model and controls the first computing component and the second computing component to work together to perform inference calculation tasks on the preset model based on the input data. During the inference process, the system executes matrix multiplication calculation tasks in the preset model through the in-memory computing array in the first computing component and executes nonlinear operator calculation tasks in the preset model through the nonlinear acceleration subcomponent in the second computing component.

14. The method according to claim 13, wherein, The step of controlling the first computing component and the second computing component to work together based on the input data to perform inference calculation tasks on the preset model includes: The input data is driven to the in-memory computing array through the in-memory computing control sub-component in the first computing component; Matrix multiplication is performed using the in-memory array to obtain query vector, key vector, and value vector; The attention weights are obtained by performing matrix multiplication based on the query vector and the key vector through the vector operation unit in the second computing component. The nonlinear acceleration subcomponent in the second computing component determines the attention operation result based on the attention weights and the value vector, and determines the output result of the current attention layer of the preset model based on the attention operation result. In response to the fact that the current attention layer is not the last attention layer, the output of the current attention layer is used as the input data, and the following steps are iteratively executed: The input data is driven to the in-memory computing array through the in-memory computing control sub-component in the first computing component; Matrix multiplication is performed using the in-memory array to obtain query vector, key vector, and value vector; The attention weights are obtained by performing matrix multiplication based on the query vector and the key vector through the vector operation unit in the second computing component. The nonlinear acceleration subcomponent in the second computing component determines the attention operation result based on the attention weights and the value vector, and determines the output result of the current attention layer of the preset model based on the attention operation result. In response to the current attention layer being the last attention layer, the output of the current attention layer is determined as the inference result of the preset model.

15. An electronic device comprising: The three-dimensional integrated storage system according to any one of claims 1-12.

16. The electronic device of claim 15, further comprising: The master device is configured as follows: The application program running the preset model generates the computation task of the three-dimensional integrated in-memory computing system, calls the three-dimensional integrated in-memory computing system to execute the computation task, and completes the inference of the preset model.

Images