Three-dimensional heterogeneous integrated storage and calculation integrated processing method and device

Abstract

The invention relates to a three-dimensional heterogeneous integrated storage and calculation integrated processing method and device, and the method comprises the steps: executing a corresponding dense matrix operation on a self-attention mechanism of a Transform model through a top layer of an artificial intelligence chip, so as to obtain a corresponding operation result; transmitting the operation result to a substrate layer based on a vertical interconnection strategy so as to pre-process the operation result, and performing data routing decision and expert selection operation on the pre-processed operation result so as to obtain selection data; and transmitting the selected data to a target RRAM-CIM expert array in the calculation layer to perform parallel calculation on the selected data to obtain parallel calculation results, and converging the parallel calculation results to an aggregator of the substrate layer according to a vertical interconnection strategy to obtain final aggregated calculation data corresponding to the model, therefore, minimization of data movement and heterogeneous matching of calculation normal forms are achieved, and calculation energy efficiency, flux and expandability are greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model, in particular to a three-dimensional heterogeneous integrated storage and calculation integrated processing method and device. BACKGROUND

[0002] The existing large language model faces serious memory wall, communication wall and power wall bottlenecks in the inference and training process. In addition, in the face of exponential growth of model size, a single general-purpose computing unit (such as CPU or GPU) is inefficient in processing the dense and sparse computing tasks with different characteristics in the Transformer model.

[0003] At present, the existing technology usually connects general-purpose CPU / GPU with high-bandwidth memory (HBM) using 2.5D integration (such as CoWoS), or uses a planar storage and calculation integrated (CIM) chip for specific calculations. However, the existing technology cannot meet the efficient access and sparse routing requirements of the massive expert weights in the MoE architecture when solving the complex heterogeneous computing of Transformer and MoE models, because the 2.5D integration is limited by the interconnection density and bandwidth of the silicon interposer. Moreover, when the planar CIM chip is integrated with digital logic and high-bandwidth memory, it still needs to pass through the traditional off-chip bus, resulting in a serious data transmission bottleneck. Therefore, the existing technology mainly has the following problems: 1. Communication efficiency bottleneck: The existing architecture cannot provide sufficient bandwidth and low-latency interconnection to support high-frequency data exchange between different computing units in the MoE architecture, resulting in a serious communication wall problem.

[0004] 2. Low energy efficiency: In the traditional architecture, the energy consumption of data movement is much higher than the actual computing energy consumption. At the same time, a single general-purpose computing unit cannot efficiently handle the dense and sparse computing tasks with different characteristics in the Transformer model, resulting in low overall energy efficiency of the system.

[0005] 3. Limited scalability: In the face of exponential growth of model size, the existing hardware architecture cannot achieve linear expansion of performance through simple stacking, and cannot effectively solve the memory wall problem.

[0006] In summary, the existing large language model computing technology has low energy efficiency, poor communication efficiency and scalability, and cannot effectively minimize data movement and match heterogeneous computing paradigms, which needs to be solved. SUMMARY

[0007] The present application provides a three-dimensional heterogeneous integrated storage and calculation integrated processing method and device to solve the problems of low energy efficiency, poor communication efficiency and scalability of the existing large language model computing technology, and difficulty in effectively minimizing data movement and matching heterogeneous computing paradigms.

[0008] The first aspect of this application provides a three-dimensional heterogeneous integrated in-memory computing processing method, comprising the following steps: performing corresponding dense matrix operations on the self-attention mechanism of a pre-built converter model through an engine layer in a preset artificial intelligence chip to obtain corresponding computation results; based on a preset vertical interconnection strategy, transmitting the computation results to a preset base layer for preprocessing the computation results in the base layer, and performing preset data routing decisions and expert selection operations on the preprocessed computation results to obtain corresponding selection data; transmitting the selection data to a corresponding target expert array in a preset computing layer for parallel computation of the selection data to obtain corresponding parallel computation results, and aggregating the parallel computation results to an aggregator in the base layer according to the vertical interconnection strategy to obtain the final aggregated computation data corresponding to the converter model.

[0009] Optionally, in one embodiment of this application, before performing the corresponding dense matrix operation on the self-attention mechanism of the pre-built converter model through the engine layer in the preset artificial intelligence chip, the method further includes: constructing a corresponding computing engine based on a preset plurality of computing acceleration units and matrix multiplication units, and vertically interconnecting the computing engine with a preset three-dimensional storage array to establish the engine layer; constructing the computing layer based on a preset local routing network and a plurality of parallel expert macro units, and constructing the base layer using a preset global controller, a hierarchical on-chip network, an aggregator, and a plurality of hybrid expert model digital functional units; and vertically stacking the base layer, the computing layer, and the engine layer from bottom to top based on a preset hybrid bonding technology to construct the artificial intelligence chip.

[0010] Optionally, in one embodiment of this application, the step of performing a corresponding dense matrix operation on the self-attention mechanism of a pre-built converter model through the engine layer in a preset artificial intelligence chip to obtain the corresponding operation result includes: obtaining the input sequence, intermediate matrix and weight data corresponding to the converter model through the engine layer, and performing a dense matrix operation corresponding to the self-attention mechanism based on the input sequence, the intermediate matrix and the weight data to obtain the operation result.

[0011] Optionally, in one embodiment of this application, the step of passing the selected data to the target expert array corresponding to a preset computing layer to perform parallel computing on the selected data to obtain the corresponding parallel computing results, and aggregating the parallel computing results to the aggregator of the base layer according to the vertical interconnection strategy to obtain the final aggregated computing data corresponding to the converter model, includes: performing sparse matrix-vector multiplication operations corresponding to the selected data in the computing layer to obtain the parallel computing results; and integrating the parallel computing results through the aggregator of the base layer to obtain the final aggregated computing data.

[0012] A second aspect of this application provides a three-dimensional heterogeneous integrated in-memory computing processing device, comprising: a matrix operation module, used to perform corresponding dense matrix operation operations on the self-attention mechanism of a pre-built converter model through an engine layer in a preset artificial intelligence chip, to obtain corresponding operation results; an expert selection module, used to transmit the operation results to a preset base layer based on a preset vertical interconnection strategy, to preprocess the operation results in the base layer, and to perform preset data routing decisions and expert selection operations on the preprocessed operation results to obtain corresponding selection data; and an aggregation calculation module, used to transmit the selection data to a corresponding target expert array in a preset calculation layer, to perform parallel calculation on the selection data, to obtain corresponding parallel calculation results, and to aggregate the parallel calculation results to an aggregator in the base layer according to the vertical interconnection strategy, to obtain the final aggregate calculation data corresponding to the converter model.

[0013] Optionally, in one embodiment of this application, it further includes: a vertical interconnect module, used to construct a corresponding computing engine based on a preset plurality of computing acceleration units and matrix multiplication units before performing corresponding dense matrix operation operations on the self-attention mechanism of the pre-built converter model through the engine layer in the preset artificial intelligence chip, and to vertically interconnect the computing engine and the preset three-dimensional storage array to establish the engine layer; a component module, used to construct the computing layer based on a preset local routing network and a plurality of parallel expert macro units, and to construct the base layer using a preset global controller, a hierarchical on-chip network, an aggregator and a plurality of hybrid expert model digital functional units; and a vertical stacking module, used to vertically stack the base layer, the computing layer and the engine layer from bottom to top based on a preset hybrid bonding technology to construct the artificial intelligence chip.

[0014] Optionally, in one embodiment of this application, the matrix operation module includes: a first execution unit, configured to obtain the input sequence, intermediate matrix and weight data corresponding to the converter model through the engine layer, and perform dense matrix operation operation corresponding to the self-attention mechanism based on the input sequence, the intermediate matrix and the weight data to obtain the operation result.

[0015] Optionally, in one embodiment of this application, the aggregation calculation module includes: a second execution unit, configured to perform sparse matrix-vector multiplication operations corresponding to the selected data in the calculation layer to obtain the parallel calculation results; and an integration unit, configured to integrate the parallel calculation results through the aggregator of the base layer to obtain the final aggregation calculation data.

[0016] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional heterogeneous integrated in-memory computing processing method as described in the above embodiments.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described three-dimensional heterogeneous integrated storage and computing method.

[0018] A fifth aspect of this application provides a computer program product, including a computer program that is executed to implement the above-described three-dimensional heterogeneous integrated in-memory computing processing method.

[0019] Therefore, the embodiments of this application have the following beneficial effects: The embodiments of this application can perform corresponding dense matrix operations on the self-attention mechanism of a pre-built Transformer model through a DRAM-PIM layer in a preset artificial intelligence chip to obtain corresponding calculation results. Based on a preset vertical interconnect strategy, the calculation results are passed to a preset base layer for preprocessing. Preset data routing decisions and expert selection operations are then performed on the preprocessed results to obtain corresponding selection data. The selection data is then passed to the corresponding target RRAM-CIM expert array in a preset computing layer for parallel computation to obtain corresponding parallel computation results. Based on the vertical interconnect strategy, the parallel computation results are aggregated to the aggregator in the base layer to obtain the final aggregated computation data corresponding to the Transformer model. This application achieves the minimization of data movement and heterogeneous matching of computational paradigms through deep three-dimensional heterogeneous integration, thereby significantly improving computational energy efficiency, throughput, and scalability. This solves the problems of low energy efficiency, poor communication efficiency and scalability, and difficulty in effectively minimizing data movement and heterogeneous matching of computational paradigms in existing large language model computation technologies.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a three-dimensional heterogeneous integrated in-memory computing processing method provided according to an embodiment of this application; Figure 2 A schematic diagram of the structure of a three-dimensional heterogeneous integrated in-memory computing processor is provided for one embodiment of this application; Figure 3 A schematic diagram of a DRAM-PIM layer structure is provided for one embodiment of this application; Figure 4 A schematic diagram of the execution logic of a three-dimensional heterogeneous integrated in-memory computing processing method provided for one embodiment of this application; Figure 5 A schematic diagram of an RRAM in-memory computing layer structure is provided for one embodiment of this application; Figure 6 A schematic diagram of a digital control and interconnection substrate structure is provided for one embodiment of this application; Figure 7 This is an example diagram of a three-dimensional heterogeneous integrated in-memory computing device according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0022] Among them, 10-three-dimensional heterogeneous integrated in-memory computing device; 100-matrix operation module, 200-expert selection module, 300-aggregation computing module; 801-memory, 802-processor, 803-communication interface. Detailed Implementation

[0023] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0024] The following description, with reference to the accompanying drawings, describes a three-dimensional heterogeneous integrated in-memory computing processing method and apparatus according to embodiments of this application. Addressing the problems mentioned in the background art, this application provides a three-dimensional heterogeneous integrated in-memory computing processing method. In this method, a pre-built Transformer model's self-attention mechanism is subjected to dense matrix operations through a DRAM-PIM layer in a pre-defined artificial intelligence chip to obtain corresponding computation results. Based on a pre-defined vertical interconnect strategy, the computation results are transmitted to a pre-defined base layer for preprocessing. The pre-processed results undergo pre-defined data routing decisions and expert selection operations to obtain corresponding selection data. The selection data is then transmitted to a target RRAM-CIM expert array in a pre-defined computing layer for parallel computation, yielding corresponding parallel computation results. These parallel computation results are then aggregated to an aggregator in the base layer according to the vertical interconnect strategy to obtain the final aggregated computation data corresponding to the Transformer model. This application, through deep three-dimensional heterogeneous integration, minimizes data movement and achieves heterogeneous matching of computational paradigms, thereby significantly improving computational energy efficiency, throughput, and scalability. This solves the problems of low energy efficiency, poor communication efficiency and scalability of existing large language model computing technologies, as well as the difficulty in effectively minimizing data movement and heterogeneous matching of computing paradigms.

[0025] Specifically, Figure 1 A flowchart illustrating a three-dimensional heterogeneous integrated in-memory computing processing method provided in an embodiment of this application.

[0026] like Figure 1 As shown, the three-dimensional heterogeneous integrated in-memory computing processing method includes the following steps: In step S101, the corresponding dense matrix operation is performed on the self-attention mechanism of the pre-built Transformer model (i.e., the converter model) through the preset DRAM-PIM layer to obtain the corresponding operation result. The corresponding dense matrix operation is performed on the self-attention mechanism of the pre-built converter model through the engine layer in the preset artificial intelligence chip to obtain the corresponding operation result.

[0027] The embodiments of this application can first complete the closed-loop calculation of Self-Attention in the near-memory computing unit of the top layer (i.e., the engine layer, or DRAM-PIM layer) in the artificial intelligence chip, thereby obtaining the corresponding calculation result.

[0028] Optionally, in one embodiment of this application, before performing the corresponding dense matrix operation on the self-attention mechanism of the pre-built converter model through the engine layer in the preset artificial intelligence chip, the method further includes: constructing a corresponding computing engine based on a preset plurality of computing acceleration units and matrix multiplication units, and vertically interconnecting the computing engine and the preset three-dimensional storage array to establish an engine layer; constructing a computing layer based on a preset local routing network and a plurality of parallel expert macrocells, and constructing a base layer using a preset global controller, a hierarchical on-chip network, an aggregator, and a plurality of hybrid expert model digital functional units; and vertically stacking the base layer, computing layer, and engine layer from bottom to top based on a preset hybrid bonding technology to construct an artificial intelligence chip.

[0029] This application embodiment can construct a corresponding three-dimensional heterogeneous integrated in-memory computing processor based on the three-dimensional heterogeneous integrated in-memory computing processing method. This processor can be used for large-scale language model computation. In the embodiments of this application, the processor can adopt a three-dimensional heterogeneous integrated architecture. Specifically, as... Figure 2 As shown, the processor includes: a top-level active high-bandwidth memory chip for accommodating large-scale data and performing near-memory computation; a middle-level multi-layer RRAM in-memory computing chip for storage and energy-efficient execution of model weight calculation; and a bottom-level logic control chip for task scheduling and data routing. The three-dimensional stacked chips are vertically interconnected through hybrid bonding technology to provide ultra-high-density, ultra-high-bandwidth data channels.

[0030] Therefore, the embodiments of this application construct a three-dimensional heterogeneous integrated in-memory processor for MOE LLM, the core of which consists of three chips with different functions, each adapted to different computing stages of LLM, and vertically stacked through hybrid bonding technology to construct an artificial intelligence chip.

[0031] It is understood that the embodiments of this application, through a modular 3D stacking design, allow for flexible addition of CIM layers according to the model size, achieving a match between the hardware architecture and the model size, and providing unprecedented scalability.

[0032] Optionally, in one embodiment of this application, the corresponding dense matrix operation is performed on the self-attention mechanism of the pre-built converter model through the engine layer in the preset artificial intelligence chip to obtain the corresponding operation result. This includes: obtaining the input sequence, intermediate matrix and weight data corresponding to the converter model through the engine layer, and performing the dense matrix operation operation corresponding to the self-attention mechanism based on the input sequence, intermediate matrix and the weight data to obtain the operation result.

[0033] It should be noted that the DRAM PIM layer in the embodiments of this application integrates high-density 3D DRAM and a digital computing engine for in-memory computation. It is responsible for storing the input sequence, a large-scale intermediate matrix (Q, K, V), and some weights, and specifically performs dense matrix operations for self-attention. Figure 3 As shown, this layer integrates the 3D DRAM memory array and the PIM engine on the same chip via high-density vertical interconnects (such as microbumps or hybrid bonding). The PIM engine (i.e., the computation engine) contains multiple dedicated MAC / MatMul units (i.e., computation acceleration units and matrix multiplication units) for matrix multiplication, located on the base logic layer of the DRAM banks. This design brings the computation logic and memory cells extremely close physically, enabling ultra-high bandwidth and ultra-low latency memory access.

[0034] In step S102, based on a preset vertical interconnection strategy, the calculation results are transmitted to a preset base layer for preprocessing. The preprocessed calculation results are then subjected to preset data routing decisions and expert selection operations to obtain the corresponding selection data.

[0035] In step S103, the selected data is passed to the target expert array in the preset computing layer to perform parallel computing on the selected data, obtain the corresponding parallel computing results, and aggregate the parallel computing results to the aggregator of the base layer according to the vertical interconnection strategy to obtain the final aggregate computing data corresponding to the converter model.

[0036] After that, as Figure 4 As shown, embodiments of this application can send the computation results to the lower layer for preprocessing via vertical interconnect. The processed data is then routed to a specific RRAM-CIM expert array in the middle layer for parallel computation via a global network and a Top-K selector. Finally, the outputs of each expert are converged to the lower layer aggregator via vertical interconnect to complete the computation of this layer.

[0037] Therefore, the embodiments of this application fundamentally eliminate memory walls and communication walls by physically integrating computation and storage in three-dimensional space, thereby minimizing data movement and significantly improving computational energy efficiency and throughput.

[0038] Optionally, in one embodiment of this application, the selected data is passed to the target expert array corresponding to the preset computing layer to perform parallel computing on the selected data, obtain the corresponding parallel computing results, and aggregate the parallel computing results to the aggregator of the base layer according to the vertical interconnection strategy to obtain the final aggregate computing data corresponding to the converter model. This includes: in the computing layer, performing sparse matrix-vector multiplication operations corresponding to the selected data to obtain parallel computing results; and integrating the parallel computing results through the aggregator of the base layer to obtain the final aggregate computing data.

[0039] In the specific implementation process, the hardware architecture of this application embodiment mainly includes: a global controller at the bottom layer, used to make data routing decisions based on sparse activation results; multiple in-memory computing expert arrays at the middle layer that can be activated in parallel, used to execute the selected computing tasks; and an aggregator at the bottom layer, used to aggregate simulation computing results from different experts; the architecture realizes accurate data delivery from the bottom layer controller to the middle layer experts through vertical interconnection, and quickly completes result integration through the hardware aggregator, thereby achieving efficient parallelism and energy consumption optimization of MoE computing.

[0040] In this embodiment of the artificial intelligence chip, the middle layer, namely the RRAM in-memory computing layer, consists of multiple parallel MoE expert macrocells. Each macrocell includes a large-scale RRAM cross-connect array, a high-speed ADC / DAC, and a lightweight local routing network. This layer stores and performs sparse matrix-vector multiplication of MoE FFN with extremely high energy efficiency and physical density. It should be noted that the middle layer is not necessarily a single layer. When the number of experts is large, multiple RRAM in-memory computing layers can be vertically stacked and interconnected via hybrid bonding, as shown in the specific structure below. Figure 5 As shown.

[0041] The underlying layer of the artificial intelligence chip in this application embodiment, namely the digital control and interconnect substrate, adopts advanced logic technology and integrates a global controller, a hierarchical network-on-chip (NoC), and dedicated digital hardware for MoE expert selection and result aggregation. For example... Figure 6As shown, this layer is the control core of the entire system. Its global controller is responsible for parsing instructions, task scheduling, and cross-layer coordination. The global on-chip network (NoC) provides high-bandwidth interconnects for data routing between the three layers and with external systems (via high-speed I / O interfaces such as CXL / PCIe). This layer also integrates several dedicated digital functional units specifically for MoE acceleration, such as a Top-K selector for fast sorting and expert selection.

[0042] It should be noted that the sparse matrix-vector multiplication in MoE FFN is processed on the in-memory computing array of the middle-layer chip, and the expert selection and result aggregation of MoE are accelerated in the dedicated hardware of the bottom chip. In this way, efficient collaboration between different computing layers is achieved through vertical data flow to maximize computing energy efficiency.

[0043] Therefore, the embodiments of this application provide MoE experts with a high-efficiency hardware implementation through RRAM-CIM technology, and solve the communication and latency problems in sparse computing through dedicated hierarchical routing and hardware aggregators.

[0044] According to the three-dimensional heterogeneous integrated in-memory computing method proposed in this application, dense matrix operations are performed on the self-attention mechanism of a pre-built Transformer model through a DRAM-PIM layer in a preset artificial intelligence chip to obtain the corresponding computation results. Based on a preset vertical interconnect strategy, the computation results are passed to a preset base layer for preprocessing. Preset data routing decisions and expert selection operations are then performed on the preprocessed results to obtain the corresponding selection data. The selection data is then passed to the corresponding target RRAM-CIM expert array in a preset computing layer for parallel computation to obtain the corresponding parallel computation results. Finally, according to the vertical interconnect strategy, the parallel computation results are aggregated to the aggregator in the base layer to obtain the final aggregated computation data corresponding to the Transformer model. This application achieves minimized data movement and heterogeneous matching of computational paradigms through deep three-dimensional heterogeneous integration, thereby significantly improving computational energy efficiency, throughput, and scalability.

[0045] Secondly, the three-dimensional heterogeneous integrated in-memory computing device proposed according to the embodiments of this application is described with reference to the accompanying drawings.

[0046] Figure 7 This is a block diagram of a three-dimensional heterogeneous integrated in-memory computing device according to an embodiment of this application.

[0047] like Figure 7 As shown, the three-dimensional heterogeneous integrated in-memory computing device 10 includes: a matrix operation module 100, an expert selection module 200, and an aggregation calculation module 300.

[0048] The matrix operation module 100 is used to perform corresponding dense matrix operation on the self-attention mechanism of the pre-built converter model through the engine layer in the preset artificial intelligence chip, so as to obtain the corresponding operation result.

[0049] The expert selection module 200 is used to transmit the calculation results to a preset base layer based on a preset vertical interconnection strategy, so that the calculation results are preprocessed in the base layer, and preset data routing decisions and expert selection operations are performed on the preprocessed calculation results to obtain the corresponding selection data.

[0050] The aggregation computing module 300 is used to pass the selected data to the corresponding target expert array in the preset computing layer to perform parallel computing on the selected data, obtain the corresponding parallel computing results, and aggregate the parallel computing results to the aggregator in the base layer according to the vertical interconnection strategy to obtain the final aggregation computing data corresponding to the converter model.

[0051] Optionally, in one embodiment of this application, the three-dimensional heterogeneous integrated in-memory computing processing device 10 of this application embodiment further includes: a vertical interconnect module, a component module, and a vertical stacking module.

[0052] The vertical interconnect module is used to construct a corresponding computing engine based on multiple preset computing acceleration units and matrix multiplication units before performing corresponding dense matrix operations on the self-attention mechanism of the pre-built converter model through the engine layer in the preset artificial intelligence chip. The computing engine and the preset three-dimensional storage array are vertically interconnected to establish the engine layer.

[0053] The module is used to build a computing layer based on a preset local routing network and multiple parallel expert macro units, and to build a base layer using a preset global controller, hierarchical on-chip network, aggregator and multiple hybrid expert model digital functional units.

[0054] Vertical stacking modules are used to vertically stack the base layer, computing layer, and engine layer from bottom to top based on a preset hybrid bonding technology to build artificial intelligence chips.

[0055] Optionally, in one embodiment of this application, the matrix operation module 100 includes: a first execution unit, configured to obtain the input sequence, intermediate matrix and weight data corresponding to the converter model through the engine layer, and perform dense matrix operation operation corresponding to the self-attention mechanism based on the input sequence, intermediate matrix and weight data to obtain the operation result.

[0056] Optionally, in one embodiment of this application, the aggregation computing module 300 includes a second execution unit and an integration unit.

[0057] The second execution unit is used in the computation layer to perform sparse matrix-vector multiplication operations corresponding to the selected data in order to obtain parallel computation results.

[0058] The integration unit is used to integrate the results of parallel computing through the aggregator in the base layer to obtain the final aggregated computing data.

[0059] It should be noted that the foregoing explanation of the embodiment of the three-dimensional heterogeneous integrated in-memory computing processing method also applies to the three-dimensional heterogeneous integrated in-memory computing processing device of this embodiment, and will not be repeated here.

[0060] The three-dimensional heterogeneous integrated in-memory computing device proposed in this application includes a matrix operation module 100, which performs dense matrix operations on the self-attention mechanism of a pre-built Transformer model through a DRAM-PIM layer in a preset artificial intelligence chip to obtain corresponding operation results; an expert selection module 200, which transmits the operation results to a preset base layer based on a preset vertical interconnection strategy, performs preprocessing on the operation results in the base layer, and performs preset data routing decisions and expert selection operations on the preprocessed operation results to obtain corresponding selection data; and an aggregation computing module 300, which transmits the selection data to the corresponding target RRAM-CIM expert array in a preset computing layer to perform parallel computing on the selection data to obtain corresponding parallel computing results, and aggregates the parallel computing results to the aggregator in the base layer according to the vertical interconnection strategy to obtain the final aggregate computing data corresponding to the Transformer model. This application achieves the minimization of data movement and heterogeneous matching of computing paradigms through deep three-dimensional heterogeneous integration, thereby significantly improving computing energy efficiency, throughput, and scalability.

[0061] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0062] When the processor 802 executes the program, it implements the three-dimensional heterogeneous integrated in-memory computing processing method provided in the above embodiments.

[0063] Furthermore, electronic devices also include: Communication interface 803 is used for communication between memory 801 and processor 802.

[0064] The memory 801 is used to store computer programs that can run on the processor 802.

[0065] The memory 801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0066] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0067] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0068] The processor 802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0069] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described three-dimensional heterogeneous integrated in-memory computing processing method.

[0070] This application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-described three-dimensional heterogeneous integrated in-memory computing processing method.

[0071] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0072] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0073] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0074] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0075] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0076] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0077] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0078] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A three-dimensional heterogeneous integrated in-memory computing processing method, characterized in that, Includes the following steps: The engine layer in the pre-built artificial intelligence chip performs corresponding dense matrix operations on the self-attention mechanism of the pre-built converter model to obtain the corresponding calculation results. Based on a preset vertical interconnection strategy, the calculation results are transmitted to a preset base layer for preprocessing. The preprocessed calculation results are then subjected to preset data routing decisions and expert selection operations to obtain the corresponding selection data. The selected data is passed to the target expert array in the preset computing layer to perform parallel computing on the selected data, obtain the corresponding parallel computing results, and aggregate the parallel computing results to the aggregator of the base layer according to the vertical interconnection strategy to obtain the final aggregate computing data corresponding to the converter model.

2. The method according to claim 1, characterized in that, Before performing the corresponding dense matrix operation on the self-attention mechanism of the pre-built converter model through the engine layer in the preset artificial intelligence chip, the following is also included: Based on a number of preset computation acceleration units and matrix multiplication units, a corresponding computing engine is constructed, and the computing engine and the preset three-dimensional storage array are vertically interconnected to establish the engine layer. The computing layer is constructed based on a preset local routing network and multiple parallel expert macro units, and the base layer is constructed using a preset global controller, hierarchical on-chip network, aggregator and multiple hybrid expert model digital functional units. Based on a preset hybrid bonding technology, the base layer, the computing layer, and the engine layer are vertically stacked from bottom to top to construct the artificial intelligence chip.

3. The method according to claim 1, characterized in that, The process involves performing dense matrix operations on the self-attention mechanism of a pre-built converter model through the engine layer in a pre-defined artificial intelligence chip to obtain the corresponding computational results, including: The engine layer obtains the input sequence, intermediate matrix, and weight data corresponding to the converter model, and performs dense matrix operation corresponding to the self-attention mechanism based on the input sequence, intermediate matrix, and weight data to obtain the operation result.

4. The method according to claim 2, characterized in that, The step of passing the selected data to the corresponding target expert array in the preset computing layer for parallel computing to obtain the corresponding parallel computing results, and then aggregating the parallel computing results to the aggregator in the base layer according to the vertical interconnection strategy to obtain the final aggregated computing data corresponding to the converter model, includes: In the computation layer, sparse matrix-vector multiplication operations corresponding to the selected data are performed to obtain the parallel computation results; The parallel computation results are integrated by the aggregator in the base layer to obtain the final aggregated computation data.

5. A three-dimensional heterogeneous integrated in-memory computing processing device, characterized in that, include: The matrix operation module is used to perform corresponding dense matrix operation operations on the self-attention mechanism of the pre-built converter model through the engine layer in the preset artificial intelligence chip, so as to obtain the corresponding operation results; The expert selection module is used to transmit the calculation results to a preset base layer based on a preset vertical interconnection strategy, so as to preprocess the calculation results in the base layer, and perform preset data routing decisions and expert selection operations on the preprocessed calculation results to obtain corresponding selection data. The aggregation computing module is used to pass the selected data to the corresponding target expert array in the preset computing layer to perform parallel computing on the selected data, obtain the corresponding parallel computing results, and aggregate the parallel computing results to the aggregator of the base layer according to the vertical interconnection strategy to obtain the final aggregation computing data corresponding to the converter model.

6. The apparatus according to claim 5, characterized in that, Also includes: The vertical interconnect module is used to construct a corresponding computing engine based on multiple preset computing acceleration units and matrix multiplication units before performing corresponding dense matrix operation operations on the self-attention mechanism of the pre-built converter model through the engine layer in the preset artificial intelligence chip, and to vertically interconnect the computing engine with the preset three-dimensional storage array to establish the engine layer. The module is used to build the computing layer based on a preset local routing network and multiple parallel expert macro units, and to construct the base layer using a preset global controller, hierarchical on-chip network, aggregator and multiple hybrid expert model digital functional units. A vertical stacking module is used to vertically stack the base layer, the computing layer and the engine layer from bottom to top based on a preset hybrid bonding technology to construct the artificial intelligence chip.

7. The apparatus according to claim 5, characterized in that, The matrix operation module includes: The first execution unit is configured to obtain the input sequence, intermediate matrix, and weight data corresponding to the converter model through the engine layer, and perform dense matrix operation corresponding to the self-attention mechanism based on the input sequence, the intermediate matrix, and the weight data to obtain the operation result.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional heterogeneous integrated in-memory computing processing method as described in any one of claims 1-4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the three-dimensional heterogeneous integrated in-memory computing processing method as described in any one of claims 1-4.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the three-dimensional heterogeneous integrated in-memory computing processing method as described in any one of claims 1-4.

Images