A selectable rom and 3d-dram unified memory architecture for large model decoding, chip and electronic device

Abstract

The application discloses a selectable ROM and 3D-DRAM unified storage architecture, a chip and electronic equipment facing large model decoding, and belongs to the technical field of integrated circuits. The application comprises a shared advanced process Logic Die and a selectable model storage backend. The shared advanced process Logic Die is integrated with a unified model storage controller, a unified model storage interface and at least one computing array. The computing array is not redesigned according to the change of the model storage backend type, and the unified model storage controller and the unified model storage interface complete medium difference conversion. On the basis of the shared advanced process Logic Die, the application selects a model weight ROM, bonds a 3D-DRAM or simultaneously sets the ROM and the 3D-DRAM through product model, and realizes transparent access of the computing array to different storage media through the unified controller, so that high energy efficiency, high bandwidth and deployment flexibility are considered.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit technology, and in particular to a unified storage architecture, chip, and electronic device for selectable ROM and 3D-DRAM for large model decoding. Background Technology

[0002] The inference process of a Large Language Model (LLM) is typically divided into two stages: Prefill and Decode. The Prefill stage processes input prompts, and its computational efficiency directly determines the latency of the first token. The Decode stage, on the other hand, generates output tokens sequentially through an autoregressive approach. Its performance—including latency between adjacent tokens, generation speed per request, system-level token throughput, and energy consumption per token—constitutes the core optimization objective of an LLM inference system.

[0003] During the decoding phase, when the system is under small or medium batch load, each decoding step generates only one or a small number of tokens for each active sequence. At this time, model weights need to be repeatedly read from the storage medium to the computing unit at each time step. However, the arithmetic intensity is usually low, causing system performance to be highly limited by effective storage bandwidth and access latency, exhibiting a "memory wall" bottleneck. As the batch size increases, weights can be reused across multiple requests within a batch, and the bottleneck gradually shifts to computing array utilization, on-chip interconnect bandwidth, data format conversion overhead, and scheduling software stack, demonstrating the complex coupling relationship between the storage and computing subsystems.

[0004] To address the aforementioned storage bottlenecks, the industry has explored various high-bandwidth storage solutions. Static Random Access Memory (SRAM) boasts extremely high read speeds, but its large cell area and low density make it difficult to fully store model weights with billions to tens of billions of parameters within chip area constraints. Dynamic Random Access Memory (DRAM) offers higher density, but traditional off-chip DRAM or High Bandwidth Memory (HBM) is limited by the number of package I / O pins, power budget, and interface bandwidth, making it difficult to meet the bandwidth requirements for large-scale weight reads and concurrent key-value cache (KV Cache) access. In recent years, 3D integration technology has provided a new path to solve this problem. By integrating logic dies with DRAM dies through wafer-to-wafer or chip-to-wafer hybrid bonding, the number of vertical interconnects can be significantly increased. However, when model weights and long-context KV Caches grow simultaneously, 3D-DRAM still needs to handle all weight reads and KV reads and writes, and capacity, bank parallelism, power consumption, and thermal constraints may still become bottlenecks.

[0005] Unlike the DRAM approach, another approach embeds model weights in a mask-ROM or a ROM-based computing array, creating a dedicated inference chip that is either a "model as a chip" or a "model as the computer." This type of ROM requires no refresh, has high density, and can reduce weight movement through wide interfaces or in-memory computing. However, the model weights are difficult to change once manufactured; when the underlying model is upgraded, the quantization method changes, LoRA / Adapter is added, or different models need to be deployed, the chip usually needs to be remanufactured.

[0006] Furthermore, in the field of structured ASICs or gate arrays, there are technologies that allow for customer customization using prefabricated base wafers and small amounts of top-layer metal or via masks. However, existing solutions remain at the level of logic function customization and do not address the unified design of weighted storage media selection and computation array access paths in LLM inference.

[0007] In summary, there is currently a lack of a unified architecture that can balance large-capacity flexible storage (adapting to long context and dynamic KVCache) with high-energy-efficiency weighted reading (adapting to the memory-intensive characteristics of Decode access), and support different storage media combinations through later customization on the same hardware platform. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention achieves a unified product model that is compatible with ROM-type, DRAM-type, and ROM+DRAM hybrid products by sharing an advanced process logic die. This allows for the creation of multiple product models targeting low latency, throughput priority, or energy efficiency priority without redesigning the main computing logic. Simultaneously, the computing array is decoupled from the specific storage medium through a memory controller, ensuring that ROM basic weights, DRAM variable weights, LoRA / Adapter increments, KV Cache, and SRAM hotspot data reside in a unified logical address space. Furthermore, static weight streams and dynamic KV streams are scheduled in parallel during the decoding phase, improving computing array utilization and reducing latency and data migration energy consumption.

[0009] To achieve the above objectives, the present invention provides a unified storage architecture for selectable ROM and 3D-DRAM for large model decoding, comprising: a shared advanced process Logic Die and selectable model storage backends, wherein the shared advanced process Logic Die integrates a unified model storage controller, a unified model storage interface and at least one computing array; the computing arrays are not redesigned with changes in the type of model storage backends, but media difference conversion is completed by the unified model storage controller and the unified model storage interface; The shared advanced process Logic Die serves as a computing and control logic carrier used by products with different storage configurations. It connects the computing array to the unified model storage controller through on-chip interconnects and connects the unified model storage controller to the selectable model storage backend through the unified model storage interface. The computing array is used to perform one or more of the following operations during the large model decoding process: matrix operation, vector operation, attention operation, feedforward neural network operation, normalization operation, activation operation, positional encoding operation, and sampling operation, and to issue model data access requests. The unified model storage controller, as the core of access mapping and scheduling, is used to receive model data access requests from the computing array, select storage media, map the model data access requests to selectable model storage backends to form unified logical requests based on model storage descriptors, and return data in a unified format to the computing array. The unified model storage interface serves as a structural channel between the unified model storage controller and the selectable model storage backend. It is used to receive unified logical requests after the storage medium is selected, and to perform data transmission and reception according to the model storage descriptor and a preset communication protocol. The selectable model storage backend includes at least three scenarios: a fixed weight backend, a dynamic storage backend, or a hybrid backend of fixed weight backend and dynamic storage backend, for data storage.

[0010] Furthermore, one or more computing arrays are connected to the unified model storage controller via on-chip network, bus, crossbar switch or hierarchical interconnect structure. The computing arrays issue model data access requests in units of tensor identifier, model layer identifier, data block identifier, logical address, access length, data format, access type and priority. During the decoding process, while processing the current model layer or the current data block, the computing array can request the unified model storage controller to prefetch the next model layer or the next data block. The unified model storage controller then coordinates the static weight stream of the fixed weight backend, the dynamic KV stream of the dynamic storage backend, and the hot data stream of the on-chip high-speed backend, so that the data supply process overlaps with the computing process.

[0011] Furthermore, the unified model storage controller specifically comprises: A model storage descriptor is pre-cached, which is used to record one or more of the following: logical address range, tensor attributes, target medium, physical starting address, data format, compression method, incremental or overlay relationship, version information, and access permissions. It receives model data access requests from one or more computing arrays, selects storage media based on tensor identifier, model layer identifier, logical address and storage descriptor, and performs reception, arbitration, queuing, merging, splitting, address translation and priority scheduling on the received model data access requests based on the model storage descriptor, and obtains a unified logical request composed of transmission logical address, access type, data length, data identifier, read / write data and access status and sends it to the unified model storage interface. It receives read data, write completion status, verification results, busy / idle status, temperature or fault information returned by the selectable model storage backend, and performs error verification, decompression, format conversion, bit width conversion, rearrangement, incremental merging or overwrite selection. It returns model weights, KV cache, model increment, activation data or intermediate computation data in a unified format to the computing array, as well as access completion or abnormal status.

[0012] Furthermore, the unified model storage interface can adopt a unified logical protocol or multiple physical ports corresponding to various media in the selectable model storage backend.

[0013] Furthermore, the unified model storage interface includes a logical interface layer, an adaptation layer, and a physical interface layer; The logic interface layer includes a unified model storage controller side and an optional model storage backend side. The unified model storage controller side is used to receive the unified logic request after the medium selection output by the unified model storage controller, and return read data, completion status, error status and training results to the unified model storage controller. The optional model storage backend side sends physical commands, addresses, clocks and write data to the optional model storage backend, and receives read data and backend status. The adaptation layer includes a fixed weight backend adapter and a dynamic storage backend adapter, which are used to convert the received unified logical requests into the commands, addresses, timing and data formats required by the target medium through the fixed weight backend adapter and the dynamic storage backend adapter. Physical interface layer: Provides predefined locations for power, ground, clock, command, address, read data, write data, check and training signals, as well as corresponding on-chip or vertical interconnect structures.

[0014] Furthermore, the solidified weight backend is a Mask-ROM formed on a common advanced process Logic Die, a model weight ROM formed by programming through a later metal layer or via layer, or an independent ROM Die connected to the Logic Die. It is used to store the basic model weights, quantization scales, constant tables and other static model data that are stable in version and read-only during the decoding process, and provides a wideband read-only data stream to the unified model storage controller through the solidified weight backend adapter and the unified model storage interface. The dynamic storage backend includes one or more 3D-DRAMDies connected to a common advanced process Logic Die, used to store all or part of the model weights, KV Cache, model increments, LoRA parameters, Adapter parameters, routing parameters, ROM overlay pages, and data that needs to be dynamically updated, and supports read and write access, refresh, training, and error management through a solidified weight backend adapter; The hybrid backend of solidified weights and dynamic storage is a combination of the two backends mentioned above. In this configuration, the solidified weights backend is used to store a large capacity of stable basic weights, while the dynamic storage backend is used to store dynamic KV cache, model increments, and weight data that needs to be updated or replaced. When the hybrid backend of solidified weights and dynamic storage is selected, the unified model storage controller sends different model data access requests to the solidified weights backend and the dynamic storage backend respectively according to the model storage descriptor, and selects, concatenates, adds, overwrites, or converts the data from both. The unified model storage interface is set up with access channels for the solidified weights backend and the dynamic storage backend respectively. The two channels can work independently or in parallel, enabling the unified model storage controller to read historical KV data or write newly generated KV data from the dynamic storage backend while reading model weights from the solidified weights backend.

[0015] Furthermore, when the solidified weight backend in the selectable model storage backend is a Mask-ROM formed in a common advanced process Logic Die, the unified model storage interface is connected to the Mask-ROM through on-chip metal interconnects. When the solidified weight backend and dynamic storage backend in the selectable model storage backend are independent ROM Die and 3D-DRAM Die, the unified model storage interface is connected to the corresponding storage Die through hybrid bonding structure, micro bumps, through silicon vias, redistribution layers or interposers. Different back-ends use the same or mappable port partitions, so that the unified model storage controller front-end and computing array do not need to change with the media.

[0016] Furthermore, the optional model storage backend also includes an on-chip high-speed backend, which is an on-chip SRAM or Register File, used to store the current computation data block, activation data, hot spot weights, hot spot KV data, address mapping table and model storage descriptor; the on-chip high-speed backend is connected to the unified model storage controller and computing array through on-chip interconnect, serving as a cache layer or temporary storage layer for the fixed weight backend and the dynamic storage backend.

[0017] A chip comprising the aforementioned unified storage architecture of selectable ROM and 3D-DRAM for large model decoding.

[0018] An electronic device comprising the aforementioned chip.

[0019] The beneficial effects of this invention are: Based on the shared advanced Logic Die process, this invention selects model weighted ROM, bonded 3D-DRAM, or simultaneously sets ROM and 3D-DRAM by product model, and enables the computing array to transparently access different storage media through a unified controller, thereby balancing high energy efficiency, high bandwidth, and deployment flexibility. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of a unified storage architecture for selectable ROM and 3D-DRAM for large model decoding according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of a hybrid backend configuration scenario of a fixed weight backend and a dynamic storage backend according to an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram illustrating the parallel scheduling of ROM access channels and DRAM access channels in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further explained and described below with reference to the accompanying drawings and embodiments.

[0024] like Figure 1 As shown, this embodiment of the invention provides a unified storage architecture for selectable ROM and 3D-DRAM for large model decoding, including: a shared advanced process Logic Die and selectable model storage backends. The shared advanced process Logic Die integrates a unified model storage controller, a unified model storage interface, and at least one computing array. This architecture uses the shared advanced process Logic Die as the computing and control foundation, the unified model storage controller as the access mapping and scheduling core, and the unified model storage interface to shield the differences in commands, timing, and physical connections between different media, enabling the same Logic Die to be selected later to form ROM-type, 3D-DRAM-type, or hybrid ROM and 3D-DRAM products.

[0025] A shared advanced process logic die serves as the common computing and control logic carrier for products with different memory configurations. The computing array is connected to a unified model memory controller via on-chip interconnects, and the unified model memory controller is connected to selectable model memory backends via a unified model memory interface. The computing array does not need to be redesigned based on changes in the model memory backend type; instead, the unified model memory controller and unified model memory interface handle media difference conversion. It is manufactured using predetermined advanced semiconductor processes. Its front-end transistor layer and main back-end interconnect layers remain identical or substantially identical across ROM-type, 3D-DRAM-type, and ROM / 3D-DRAM hybrid products, thereby reducing the cost of repeatedly developing advanced process logic masks for different model stability, capacity, and update requirements. Model-related differences are limited to predetermined late-stage customization layers, model memory dies, bonding steps, fuse / boot configurations, or combinations thereof.

[0026] Inputs include: model configuration, startup configuration, clock, power and control information from external chip sources or the management processor; and read data, status information and error information from the model storage backend.

[0027] Outputs include: access commands, addresses, chip select, read / write control, clock, and training signals for the model storage backend; and inference results, status, and diagnostic information for external systems.

[0028] A computation array is used to perform one or more of the following operations during the decoding process of a large model: matrix operations, vector operations, attention operations, feedforward neural network operations, normalization operations, activation operations, positional encoding operations, and sampling operations. The computation array may include one or more of the following: matrix-vector multiplication units, multiply-accumulate units, attention processing units, FFN processing units, vector processing units, local registers, local caches, and data transport units.

[0029] One or more computing arrays are connected to a unified model storage controller via an on-chip network, bus, crossbar switch or hierarchical interconnect structure. The computing arrays issue model data access requests in units of tensor identifier, model layer identifier, data block identifier, logical address, access length, data format, access type and priority. During the decoding process, while processing the current model layer or the current weight data block, the computing array can request the unified model storage controller to prefetch the next model layer or the next weight data block. The unified model storage controller coordinates the static weight stream of the solidified weight backend, the dynamic KV stream of the dynamic storage backend, and the hot spot data stream of the on-chip high-speed backend, so that the data supply process overlaps with the computing process.

[0030] Input: Base weights, effective weights, KV Cache, quantization parameters, sparse indexes, activation data, and other computational operands returned by the unified model storage controller.

[0031] Outputs include: media-independent model data access requests to the unified model storage controller, newly generated key-value data, intermediate results or activation data that needs to be written back; and calculation results for subsequent processing units.

[0032] The Unified Model Memory Controller (UMMC), as the core of access mapping and scheduling, receives model data access requests from the compute array and maps these requests to the corresponding physical storage space in ROM, 3D-DRAM, or a combination thereof, based on the model storage descriptor. It then returns data in a unified format to the compute array. The UMMC can also connect to the management processor, boot configuration circuitry, model manifest storage area, and on-chip SRAM to obtain model version, storage mapping, and fault rollback configuration. For mixed ROM and 3D-DRAM configurations, the controller can drive the ROM channel and DRAM channel separately, selecting, concatenating, adding, or overwriting data from both before sending it to the compute array. Specifically: The model storage descriptor is pre-cached. It records one or more of the following: logical address range, tensor attributes, target medium, physical starting address, data format, compression method, incremental or overwrite relationship, version information, and access permissions: (Examples of model storage descriptor fields are shown in the table below.) Table 1 It receives model data access requests from one or more computing arrays, selects storage media based on tensor identifier, model layer identifier, logical address and storage descriptor, and performs reception, arbitration, queuing, merging, splitting, address translation and priority scheduling on the received model data access requests based on the model storage descriptor. It then obtains a unified logical request consisting of transmission logical address, access type, data length, data identifier, read / write data and access status and sends it to the unified model storage interface.

[0033] It receives read data, write completion status, verification results, busy / idle status, temperature or fault information returned by the selectable model storage backend, and performs error verification, decompression, format conversion, bit width conversion, rearrangement, incremental merging or overwrite selection. It returns model weights, KV cache, model increment, activation data or intermediate computation data in a unified format to the computing array, as well as access completion or abnormal status.

[0034] Upstream input: Model data access request issued by the computing array, which may include at least tensor identifier, model layer identifier, data block identifier, logical address, access length, data format, access type and priority.

[0035] Downstream inputs: Read data returned by the model storage backend, write completion status, verification results, busy / idle status, temperature or fault information.

[0036] Upstream output includes: model weights in a unified format, key-value cache, model increments, activation data or intermediate computation data, and access completion or abnormal status.

[0037] Downstream outputs: physical address after media adaptation, read / write commands, data length, burst parameters, write data, verification information, and scheduling control information.

[0038] The Unified Model Storage Interface (UMI) serves as a structural channel between the Unified Model Storage Controller and selectable model storage backends, shielding ROM and 3D-DRAM from differences in addressing modes, command formats, read / write attributes, data bit widths, access timings, error checking methods, and physical connection forms. The UMI can manifest as a unified logical protocol or include multiple physical ports corresponding to specific media. The UMI receives unified logical requests and performs data transmission and reception according to the model storage descriptor and a preset communication protocol. The UMI comprises a logical interface layer, an adaptation layer, and a physical interface layer.

[0039] The logic interface layer includes a unified model storage controller side and an optional model storage backend side. The unified model storage controller side is used to receive the unified logic request after the medium selection output by the unified model storage controller, and return read data, completion status, error status and training results to the unified model storage controller. The optional model storage backend side sends physical commands, addresses, clocks and write data to the optional model storage backend, and receives read data and backend status.

[0040] The adaptation layer is used to convert received unified logical requests into the commands, addresses, timing and data formats required by the target medium through a fixed weight back-end adapter, a dynamic storage back-end adapter or an on-chip high-speed back-end adapter.

[0041] Physical interface layer: Provides predefined locations for power, ground, clock, command, address, read data, write data, check and training signals, as well as corresponding on-chip or vertical interconnect structures.

[0042] Controller-side inputs / outputs: Receives model data access requests and write data after medium selection, and returns read data, completion status, error status and training results to the controller.

[0043] Storage-side input / output: Sends physical commands, addresses, clock signals, and write data to ROM or 3D-DRAM, and receives read data and backend status. ROM channels can omit the write data path, while DRAM channels support both reading and writing simultaneously.

[0044] The selectable model storage backend includes three options: a fixed weight backend, a dynamic storage backend, and a hybrid backend combining fixed weight and dynamic storage. These options allow the unified model storage controller to select the appropriate media. The selectable model storage backend is used to store one or more of the following: basic weights, variable weights, model increments, key-value cache, quantization parameters, sparse indexes, expert routing information, vocabulary data, intermediate computation data, and configuration information required for large model decoding.

[0045] The embedded weights (ROM) backend can be a Mask-ROM formed on a common advanced process Logic Die, a model weights ROM formed through subsequent metal layer or via layer programming, or a standalone ROM Die connected to the Logic Die. The ROM backend is mainly used to store the basic model weights, quantization scales, constant tables, and other static model data that are version-stable and read-only or essentially read-only during decoding, and provides a wideband read-only data stream to the unified model storage controller through the ROM backend adapter and unified model storage interface.

[0046] The dynamic storage (3D-RAM) backend consists of one or more writable 3D-DRAM dies connected to a common advanced process logic die via hybrid bonding, through-silicon vias, microbumps, or other vertical interconnect structures. The dynamic storage backend stores all or part of the model weights, key-value cache, model increments, LoRA parameters, adapter parameters, routing parameters, ROM overlay pages, and data that needs to be dynamically updated. It supports read / write access, refresh, training, and error management through the dynamic storage backend adapter.

[0047] In a hybrid backend configuration combining fixed weights and dynamic storage, ROM stores a large volume of stable base weights, while 3D-DRAM stores dynamic key-value cache, model increments, and weight data that needs updating or replacement. The unified model storage controller sends different model data access requests to ROM and 3D-DRAM respectively based on the model storage descriptor, and performs selection, concatenation, addition, overwriting, or format conversion on the data from both. Figure 2 As shown.

[0048] Incremental merging method: Merge the base weight W in ROM. base The increment ΔW in the 3D-DRAM is fed into the merging unit to form the effective weight W. effectiveThe increment can represent the LoRA low-rank matrix, adapter parameters, quantization scale correction, or other model patches.

[0049] Overwrite replacement method: Based on the overwrite bitmap, overwrite page table or version descriptor, directly select replacement data in 3D-DRAM to overwrite the corresponding page, data block or tensor in ROM.

[0050] It also includes an on-chip high-speed backend, which is an on-chip SRAM or Register File used to store the current computation data block, activation data, hot spot weights, hot spot KV data, address mapping table, and model storage descriptor. The on-chip SRAM is connected to the unified model storage controller and computing array through on-chip interconnects, and can be used as a cache layer or temporary storage layer for ROM and 3D-DRAM, in conjunction with ROM and 3D-DRAM (regardless of the medium chosen, it exists in the on-chip high-speed backend).

[0051] When the media selection is for both a fixed weight backend and a dynamic storage backend, the unified model storage interface is configured with ROM access channels and DRAM access channels respectively. These two channels can work independently or in parallel, enabling the unified model storage controller to read historical key-value (KV) data from 3D-DRAM or write newly generated KV data while simultaneously reading model weights from ROM. Figure 3 As shown.

[0052] When the solidified weight backend in the selectable model storage backend is a Mask-ROM formed in a common advanced process Logic Die, the unified model storage interface is connected to the Mask-ROM via on-chip metal interconnects.

[0053] When the solidified weight backend and dynamic storage backend in the selectable model storage backend are independent ROM Die and 3D-DRAM Die, the unified model storage interface is connected to the corresponding storage Die through hybrid bonding structure, microbumps, through-silicon vias, redistribution layers or interposers.

[0054] Different back-ends use the same or mappable port partitions, so that the unified model storage controller front-end and computing array do not need to change with the media.

[0055] This invention also provides a chip, including the above-described unified storage architecture of selectable ROM and 3D-DRAM for large model decoding.

[0056] This invention also provides an electronic device, including the chip described above.

[0057] The embodiments described above are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Various modifications, equivalent substitutions, or variations made by those skilled in the art to the present invention without departing from the spirit and substance of the invention should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A unified storage architecture for selectable ROM and 3D-DRAM for large model decoding, characterized in that, include: It uses a common advanced process Logic Die and an optional model storage backend. The common advanced process Logic Die integrates a unified model storage controller, a unified model storage interface, and at least one compute array. The compute array is not redesigned with changes in the type of model storage backend, but media difference conversion is completed by the unified model storage controller and the unified model storage interface. The shared advanced process Logic Die serves as a computing and control logic carrier used by products with different storage configurations. It connects the computing array to the unified model storage controller through on-chip interconnects and connects the unified model storage controller to the selectable model storage backend through the unified model storage interface. The computing array is used to perform one or more of the following operations during the large model decoding process: matrix operation, vector operation, attention operation, feedforward neural network operation, normalization operation, activation operation, positional encoding operation, and sampling operation, and to issue model data access requests. The unified model storage controller, as the core of access mapping and scheduling, is used to receive model data access requests from the computing array, select storage media, map the model data access requests to selectable model storage backends to form unified logical requests based on model storage descriptors, and return data in a unified format to the computing array. The unified model storage interface serves as a structural channel between the unified model storage controller and the selectable model storage backend. It is used to receive unified logical requests after the storage medium is selected, and to perform data transmission and reception according to the model storage descriptor and a preset communication protocol. The selectable model storage backend includes at least three scenarios: a fixed weight backend, a dynamic storage backend, or a hybrid backend of fixed weight backend and dynamic storage backend, for data storage.

2. The selectable ROM and 3D-DRAM unified storage architecture for large model decoding as described in claim 1, characterized in that: One or more computing arrays are connected to a unified model storage controller via an on-chip network, bus, crossbar switch or hierarchical interconnect structure. The computing arrays issue model data access requests in units of tensor identifier, model layer identifier, data block identifier, logical address, access length, data format, access type and priority. During the decoding process, while processing the current model layer or the current data block, the computing array can request the unified model storage controller to prefetch the next model layer or the next data block. The unified model storage controller then coordinates the static weight stream of the fixed weight backend, the dynamic KV stream of the dynamic storage backend, and the hot data stream of the on-chip high-speed backend, so that the data supply process overlaps with the computing process.

3. The selectable ROM and 3D-DRAM unified storage architecture for large model decoding as described in claim 2, characterized in that, The unified model storage controller is specifically: A model storage descriptor is pre-cached, which is used to record one or more of the following: logical address range, tensor attributes, target medium, physical starting address, data format, compression method, incremental or overlay relationship, version information, and access permissions. It receives model data access requests from one or more computing arrays, selects storage media based on tensor identifier, model layer identifier, logical address and storage descriptor, and performs reception, arbitration, queuing, merging, splitting, address translation and priority scheduling on the received model data access requests based on the model storage descriptor, and obtains a unified logical request composed of transmission logical address, access type, data length, data identifier, read / write data and access status and sends it to the unified model storage interface. It receives read data, write completion status, verification results, busy / idle status, temperature or fault information returned by the selectable model storage backend, and performs error verification, decompression, format conversion, bit width conversion, rearrangement, incremental merging or overwrite selection. It returns model weights, KV cache, model increment, activation data or intermediate computation data in a unified format to the computing array, as well as access completion or abnormal status.

4. The selectable ROM and 3D-DRAM unified storage architecture for large model decoding as described in claim 1, characterized in that: The unified model storage interface can adopt a unified logical protocol or multiple physical ports corresponding to various media in the selectable model storage backend.

5. The selectable ROM and 3D-DRAM unified storage architecture for large model decoding according to claim 1, characterized in that: The unified model storage interface includes a logical interface layer, an adaptation layer, and a physical interface layer; The logic interface layer includes a unified model storage controller side and an optional model storage backend side. The unified model storage controller side is used to receive the unified logic request after media selection output by the unified model storage controller, and return read data, completion status, error status and training results to the unified model storage controller. The optional model storage backend side sends physical commands, addresses, clocks and write data to the optional model storage backend, and receives read data and backend status. The adaptation layer includes a fixed weight backend adapter and a dynamic storage backend adapter, which are used to convert the received unified logical requests into the commands, addresses, timing and data formats required by the target medium through the fixed weight backend adapter and the dynamic storage backend adapter. Physical interface layer: Provides predefined locations for power, ground, clock, command, address, read data, write data, check and training signals, as well as corresponding on-chip or vertical interconnect structures.

6. The selectable ROM and 3D-DRAM unified storage architecture for large model decoding according to claim 5, characterized in that: The solidified weight backend is a Mask-ROM formed on a common advanced process Logic Die, a model weight ROM formed by programming through a later metal layer or via layer, or an independent ROM Die connected to the Logic Die. It is used to store the basic model weights, quantization scales, constant tables and other static model data that are stable in version and read-only during the decoding process, and provides a wideband read-only data stream to the unified model storage controller through the solidified weight backend adapter. The dynamic storage backend includes one or more 3D-DRAM dies connected to a common advanced process Logic Die, used to store all or part of the model weights, KV Cache, model increments, LoRA parameters, Adapter parameters, routing parameters, ROM overlay pages, and data that needs to be dynamically updated, and supports read and write access, refresh, training, and error management through a solidified weight backend adapter; The hybrid backend of solidified weights and dynamic storage is a combination of the two backends mentioned above. In this configuration, the solidified weights backend is used to store a large capacity of stable basic weights, while the dynamic storage backend is used to store dynamic KV cache, model increments, and weight data that needs to be updated or replaced. When the hybrid backend of solidified weights and dynamic storage is selected, the unified model storage controller sends different model data access requests to the solidified weights backend and the dynamic storage backend respectively according to the model storage descriptor, and selects, concatenates, adds, overwrites, or converts the data from both. The unified model storage interface is set up with access channels for the solidified weights backend and the dynamic storage backend respectively. The two channels can work independently or in parallel, enabling the unified model storage controller to read historical KV data or write newly generated KV data from the dynamic storage backend while reading model weights from the solidified weights backend.

7. The selectable ROM and 3D-DRAM unified storage architecture for large model decoding as described in claim 6, characterized in that: When the solidified weight backend in the selectable model storage backend is a Mask-ROM formed in a common advanced process Logic Die, the unified model storage interface is connected to the Mask-ROM through on-chip metal interconnects. When the solidified weight backend and dynamic storage backend in the selectable model storage backend are independent ROM Die and 3D-DRAMDie, the unified model storage interface is connected to the corresponding storage Die through hybrid bonding structure, micro bumps, through silicon vias, redistribution layers or intermediary layers. Different back-ends use the same or mappable port partitions, so that the unified model storage controller front-end and computing array do not need to change with the media.

8. The selectable ROM and 3D-DRAM unified storage architecture for large model decoding according to claim 6, characterized in that: The optional model storage backend also includes an on-chip high-speed backend, which is an on-chip SRAM or Register File, used to store the current computation data block, activation data, hot spot weights, hot spot KV data, address mapping table and model storage descriptor; the on-chip high-speed backend is connected to the unified model storage controller and computing array through on-chip interconnect, and serves as a cache layer or temporary storage layer for the fixed weight backend and the dynamic storage backend.

9. A chip, characterized in that, Includes the selectable ROM and 3D-DRAM unified storage architecture for large model decoding as described in any one of claims 1-8.

10. An electronic device, characterized in that, Includes the chip described in claim 9.

Images