High bandwidth flash data stream reconstruction method, system, device and medium

Abstract

The application provides a high-bandwidth flash memory data stream reconstruction method, system, device and medium, which is applied to an HBF chip containing a substrate wafer and a NAND core layer flash memory, and the substrate wafer is integrated with a computing unit, a prefetch engine, a NAND controller and an L2 cache with a double cache area. The application starts inference by configuring parameters first, initializing the prefetch engine, and loading first layer model data in parallel through the NAND controller multi-channel; in the operation stage, the computing unit and the prefetch engine are cooperated, the double cache is alternately and circularly reused to build a computing and prefetching parallel pipeline, the smooth connection of each layer calculation and lower layer data supply is realized; after the pipeline is stabilized, the time consumption ratio of single layer calculation and lower layer prefetching is counted in real time, the prefetching data amount is adaptively adjusted to match the supply and calculation rate, the pipeline operation is stabilized, and the overall throughput efficiency of model inference is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, device and medium for reconstructing high-bandwidth flash data streams. Background Technology

[0002] Large model inference places high demands on storage capacity, bandwidth, and latency. High-bandwidth flash (HBF), as a new storage technology between HBM (High-bandwidth Memory) and SSD (Solid State Drive), uses vertically stacked NAND flash memory chips and is designed to provide large capacity and high bandwidth to support large model inference.

[0003] However, existing HBFs still rely on the traditional on-demand read mode, which has several drawbacks: serial execution of computation and data loading leads to long latency, low actual bandwidth utilization under random access, the controller lacks awareness of computational semantics, and the caching mechanism is too passive. These shortcomings limit the actual performance of HBFs, preventing them from fully realizing their theoretical advantages in large-model inference scenarios, and they cannot currently effectively replace or supplement HBMs. Therefore, an effective high-bandwidth flash data stream reconstruction method is urgently needed to solve these problems. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a high-bandwidth flash memory data stream reconstruction method, system, device and medium that overcomes or at least partially solves the above problems.

[0005] To achieve the above and other related objectives, this invention provides a high-bandwidth flash memory data stream reconstruction method, applied to an HBF chip comprising a substrate die and NAND flash memory, wherein the substrate die integrates a computing unit, a prefetch engine, a NAND controller, and an L2 cache with dual buffers. The method includes: Configure the model hardware parameters, initialize the prefetch engine, and control the NAND controller to read the first layer data of the model in parallel through multiple channels to complete the startup loading of model inference computation; During model computation, a hardware pipeline for parallel execution of computation and data prefetching is constructed based on the parallel collaboration between the computation unit and the prefetching engine, and by utilizing the alternating switching and circular reuse mechanism of the double buffers in the L2 cache, so as to achieve a smooth connection between the computation of each layer of the model and the data supply of the lower layer. Once the pipeline reaches a stable state, the ratio of the computation time of a single layer to the data prefetching time of the next layer is continuously calculated. Based on this ratio, the amount of prefetched data for subsequent layers is dynamically adjusted to match the data supply speed with the computation speed, ensuring stable pipeline operation.

[0006] Optionally, an L1 cache is also integrated on the bare die substrate. The process of configuring model hardware parameters, initializing the prefetch engine, and controlling the NAND controller to read the model's first-layer data in parallel through multiple channels to complete the startup loading of model inference computation includes: Write the model layer number, single-layer weight size, single-layer KV cache size, weight starting base address, and KV cache starting base address into the configuration register inside the bare die substrate to complete the loading and configuration of the model's basic hardware parameters; The prefetch engine is initialized, including setting the initial prefetch depth, clearing all pending tasks in the prefetch queue, and simultaneously marking the internal double buffers of the associated L2 cache as having no valid data, so that the prefetch engine is switched to the scheduling ready state. Based on the loaded weight starting base address, the storage address range of the first layer data is calculated. The prefetch engine converts the storage address range into NAND read instructions and sends them to the NAND controller. The NAND controller then reads the original weight data from the stacked NAND core flash memory through a multi-channel parallel architecture. The original weight data is sent to the L1 cache to complete ECC error correction and decoding. The processed data is stored in the first buffer of the L2 cache and marked as readable and ready. The loading of the first layer data is then completed.

[0007] Optionally, the L2 cache includes a first buffer and a second buffer. During model computation, based on the parallel collaboration between the computation unit and the prefetch engine, and utilizing the alternating switching and circular reuse mechanism of the two buffers within the L2 cache, a hardware pipeline for parallel execution of computation and data prefetching is constructed to achieve a smooth connection between the computation of each layer of the model and the data supply of the next layer, including: During the pipeline startup initialization phase, the computing unit reads the first layer data of the model from the first buffer of the L2 cache and performs self-attention mechanism operations or feedforward network operations. The prefetch engine synchronously obtains the storage address of the second layer weight data and the KV cache. After address resolution, contiguous address merging, and NAND physical address translation, it generates multi-plane parallel prefetch commands and adds them to the prefetch queue. The NAND controller loads the second layer data into the second buffer of the L2 cache through multi-channel parallel reading according to the prefetch commands in the prefetch queue, and marks the second buffer as ready, completing the pipeline initialization setup. Once the first-layer computation task is completed and the data in the second buffer is ready, the data reading path of the computation unit is switched to the second buffer, and the second-layer computation of the model is started. Simultaneously, the storage address of the third-layer data is obtained by the prefetch engine. Through address optimization and parallel prefetching processes, the third-layer data is prefetched into the vacant first buffer. This process continues to iterate in subsequent layers of the model. Relying on the working mode of alternating switching and cyclic reuse of dual buffers, the computation of each layer of the model and the prefetching of the next layer of data are completely overlapped in the time dimension, continuously realizing the continuous parallel execution of computation and data preparation, forming a stable hardware pipeline.

[0008] Optionally, the step of synchronously acquiring the storage addresses of the second-layer weight data and the KV cache by the prefetch engine, and generating multi-plane parallel prefetch commands and adding them to the prefetch queue after address resolution, contiguous address merging, and NAND physical address translation, includes: The built-in address generator of the computing unit retrieves the starting base address of the second layer weight and the starting base address of the KV cache, integrates the two types of base addresses to calculate the corresponding storage address of the second layer, and sends it to the prefetch engine. The prefetch engine parses the storage address of the second layer and compares it with the existing pending addresses in the prefetch queue one by one to verify whether the parsed storage address meets the merging condition. If the condition is met, the consecutive and overlapping addresses are merged to complete the merging; if not, the parsed storage address is retained separately. After address merging, discrimination, and processing are completed, the valid logical addresses are extracted and converted into physical addresses that are compatible with the NAND controller. The prefetch tasks are organized according to the physical address arrangement order, and multi-plane parallel prefetch commands are generated accordingly. These commands are stored in the prefetch queue and the queue task list is updated synchronously.

[0009] Optionally, the step of loading the second layer data into the second buffer of the L2 cache by the NAND controller according to the prefetch command in the prefetch queue through a multi-channel parallel reading method, and marking the second buffer as ready, includes: The NAND controller reads and parses the multi-plane parallel prefetch commands in the prefetch queue, extracts the target physical address and data length information, and distributes the read tasks to the corresponding NAND channels and logic cells. The NAND controller controls each NAND channel to synchronously read the second layer weight and KV cache data according to the assigned read task, and writes the read data into the second buffer of the L2 cache; after the data writing is completed, the second buffer is marked as ready.

[0010] Optionally, after the step of constructing a hardware pipeline for parallel execution of computation and data prefetching based on the parallel collaboration of the computing unit and the prefetching engine, and utilizing the alternating switching and circular reuse mechanism of the double buffers in the L2 cache to achieve a smooth connection between the computation of each layer of the model and the data supply of the next layer, the method further includes: The prefetch engine continuously monitors the prefetching time of data at each layer. When the prefetching time exceeds the preset time threshold, a prefetching anomaly is determined, and an anomaly signal is reported to the multi-plane scheduler built into the NAND controller, and the anomaly handling mechanism is started synchronously. After receiving an anomaly signal, the multi-plane scheduler sets the prefetch command corresponding to the timeout layer as the priority scheduling level, prioritizes the scheduling of computing and storage resources to perform data loading; and simultaneously suspends various non-urgent background tasks to concentrate resources to ensure that the data at this layer is loaded and ready in a timely manner. If the current level of computation is about to complete, but the prefetched data it depends on is not ready and the computation unit is about to enter a waiting state, the operation of the computation unit is paused and a timeout counter is started to record the waiting time. During this period, the hardware pipeline relies on the dual-buffer alternating scheduling mechanism to schedule other data-ready levels to execute computation tasks, maintaining the basic parallel operation of the pipeline. Once the timeout layer data is loaded, the computing unit resumes operation immediately, and the prefetch depth and resource allocation weight of subsequent layers are dynamically adjusted based on the timeout record. If a subsequent layer triggers a prefetch timeout multiple times, the prefetch depth is reduced level by level. If the frequency of prefetch timeouts in subsequent layers continues to decrease, the prefetch depth is increased level by level.

[0011] Optionally, dynamically adjusting the amount of prefetched data for subsequent levels based on this ratio includes: When the ratio of the computation time of a single layer to the data prefetching time of the next layer exceeds the preset high threshold, the prefetching depth is increased to load more subsequent layer data in advance. When the ratio of the computation time of a single layer to the data prefetching time of the next layer is lower than a preset low threshold, the prefetching depth is reduced to cut unnecessary data overhead. When the ratio of the computation time of a single layer to the data prefetching time of the next layer is within a reasonable range, the current prefetching depth remains unchanged; wherein, the reasonable range is between a preset high threshold and a preset low threshold.

[0012] Secondly, the present invention also provides a high-bandwidth flash memory data stream reconstruction system, applied to an HBF chip comprising a substrate die and NAND flash memory, wherein the substrate die integrates a computing unit, a prefetch engine, a NAND controller, and an L2 cache with dual buffers, the system comprising: The initialization module is used to configure the model hardware parameters, initialize the prefetch engine, and control the NAND controller to read the first layer data of the model in parallel through multiple channels to complete the system initialization and startup. The building module is used for model computation. Based on the parallel collaboration between the computing unit and the prefetching engine, and utilizing the alternating switching and circular reuse mechanism of the double buffer in the L2 cache, a hardware pipeline for parallel execution of computation and data prefetching is constructed to achieve a smooth connection between the computation of each layer of the model and the data supply of the lower layer. The adjustment module is used to continuously calculate the ratio of the computation time of a single layer to the data prefetching time of the next layer after the pipeline enters a stable state, and dynamically adjust the amount of prefetched data in subsequent layers based on this ratio to match the data supply speed with the computation speed, ensuring stable pipeline operation.

[0013] Thirdly, the present invention provides an electronic device comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program stored in the memory to cause the electronic device to perform the high-bandwidth flash data stream reconstruction method as described above.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the high-bandwidth flash data stream reconstruction method described above.

[0015] Fifthly, the present invention provides a computer program product including computer program code, which, when run on a computer, causes the computer to implement the method described above.

[0016] The above-described one or more technical solutions provided by this invention can have the following advantages or at least achieve the following technical effects: This invention first configures parameters and initializes the prefetch engine, then loads the first-layer model data in parallel via multiple channels of the NAND controller to start inference. During the computation phase, the computation unit and the prefetch engine work together, and the dual caches are used alternately and repeatedly to build a parallel pipeline for computation and prefetching, so as to achieve a smooth connection between each layer of computation and the data supply of the next layer. After the pipeline stabilizes, the ratio of the computation time of a single layer to the prefetch time of the next layer is calculated in real time, and the amount of prefetched data is adaptively adjusted to match the supply and computation rate, so as to stabilize the pipeline operation and improve the overall throughput efficiency of model inference. Attached Figure Description

[0017] Figure 1 The diagram shows a flowchart of a high-bandwidth flash memory data stream reconstruction method according to an embodiment of the present invention.

[0018] Figure 2 The diagram shows a hardware architecture schematic of the HBF chip in one embodiment of the present invention.

[0019] Figure 3 The diagram shown is a schematic diagram of an L2 cache double buffer structure in one embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram showing the stage division of the data processing pipeline in one embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of the functional modules of a high-bandwidth flash data stream reconstruction system according to an embodiment of the present invention.

[0022] Figure 6 The diagram shown is a schematic representation of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0023] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0025] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0026] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0027] Unless otherwise stated, the term "multiple" means two or more.

[0028] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0029] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0030] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0031] Please see Figure 1 An embodiment of the present invention provides a high-bandwidth flash memory data stream reconstruction method, applied to an HBF chip comprising a substrate die and NAND flash memory, wherein the substrate die integrates a computing unit, a prefetch engine, a NAND controller, and an L2 cache with dual cache areas. The method may include the following steps S10~S30: Step S10: Configure model hardware parameters, initialize the prefetch engine, and control the NAND controller to read the first layer data of the model in parallel through multiple channels to complete the startup loading of model inference computation.

[0032] The substrate integrates computing units, a prefetch engine, a NAND controller, an L1 buffer, and an L2 cache. The L1 buffer is connected to the NAND controller, the L2 cache is connected to the computing unit and has dual buffers (a first buffer and a second buffer) inside, and the prefetch engine is connected between the L1 cache and the L2 cache. Through their coordinated operation, high-bandwidth data stream scheduling and reconstruction are achieved.

[0033] The computation unit is the core processing module that performs model calculations and is responsible for completing mathematical operations in neural network layers such as self-attention mechanisms and feedforward networks.

[0034] The prefetch engine, the core module for data pre-scheduling, generates and sends out read requests in advance by predicting the data required for the next calculation, thereby hiding storage latency.

[0035] The NAND controller is a hardware module that manages the underlying storage media. It receives read requests from the prefetch engine and actually extracts data from the NAND flash memory array.

[0036] Please see Figure 2 The diagram illustrates the HBF chip hardware architecture. After the computing unit generates an address, the prefetch engine translates it into a read instruction. Parallel data transmission is achieved by the NAND controller through a multi-plane scheduler. Data is temporarily stored in layers by multi-level buffer units to ensure transmission speed. The bottom layer provides storage support through a 16-layer stacked NAND core. The entire architecture is designed for AI inference, achieving efficient data flow between computing and storage through hardware and software co-optimization.

[0037] Hardware parameters refer to the basic configuration information used to configure and initialize the operating status of the internal hardware modules of the HBF chip (such as computing units, prefetch engine, and NAND controller). They typically include the following three categories: Model and memory topology parameters, including the number of model layers, weight size, and physical address mapping table (i.e., indicating the start and end physical addresses of the first layer weights and other data in the NAND storage stack).

[0038] Prefetch and scheduling control parameters are used to configure the prefetch engine to efficiently hide the access latency of NAND flash memory; these include the initial prefetch depth, dual buffer (L1 / L2 cache) capacity configuration, and prefetch trigger threshold.

[0039] The underlying storage interface parameters are used to configure the NAND controller to ensure that the physical channel for reading underlying data is unobstructed. These parameters include NAND channel status, ECC verification mode, and timing parameters.

[0040] First-layer data refers to the basic computational data required for the first layer of model inference, which only includes the first-layer model weight parameters.

[0041] In the specific implementation, during the system initialization phase, the model hierarchical parameters and the base address of the weight and key-value pair cache are first written into the configuration register of the substrate bare die based on the hardware parameter configuration of the model. Then, the prefetch engine is reset and its prefetch queue is cleared, and the state of the L2 buffer is marked as invalid to complete the environment reset. Subsequently, the control address generator calculates the physical address range of the first layer weights, and the NAND controller is scheduled to obtain data through a multi-channel parallel reading mechanism. After the data is ECC checked by the L1 buffer, it is stored in the first storage block of the L2 buffer and its state is updated to ready, thereby completing the first connection of the data flow reconstruction link and realizing the start loading of model inference computation.

[0042] In step S20, during model computation, based on the parallel collaboration between the computation unit and the prefetch engine, and utilizing the alternating switching and circular reuse mechanism of the dual buffers in the L2 cache, a hardware pipeline for parallel execution of computation and data prefetching is constructed to achieve a smooth connection between the computation of each layer of the model and the data supply of the next layer.

[0043] The alternating switching and cyclic reuse mechanism refers to the hardware scheduling method in which the L2 cache is set up with two independent buffers that take turns to undertake working and standby tasks.

[0044] Hardware pipeline refers to the operating mechanism of executing tasks in parallel and overlapping at the hardware level.

[0045] In the specific implementation, during the pipeline initialization phase, the computation unit reads the first-layer data from the first buffer of the L2 cache to complete the first-layer Transformer computation of the model. Simultaneously, the prefetch engine completes the address resolution, merging optimization, and address translation of the second-layer weights and KV cache, generating parallel prefetch commands. Then, the NAND controller loads the second-layer data into the second buffer of the L2 cache through multi-channel parallel reading and marks it as ready. After the first-layer computation is complete and the second buffer is ready, the computation unit switches its data reading path to the second buffer, initiating the next layer computation. Simultaneously, the prefetch engine uses the idle first buffer to prefetch data for subsequent layers. This mechanism continuously iterates through the multi-layer computation of the model, ensuring that the computation of each layer and the data prefetching process of the next layer completely overlap in time, maintaining parallel execution of computation and data prefetching. This ultimately forms a continuous and stable hardware pipeline inference mechanism, achieving a smooth connection between the computation of each layer of the model and the data supply of the next layer.

[0046] Step S30: After the pipeline enters a stable state, the ratio of the computation time of a single layer to the data prefetching time of the next layer is continuously calculated, and the amount of prefetched data of subsequent layers is dynamically adjusted based on this ratio to match the data supply speed with the computation speed, so as to ensure the stable operation of the pipeline.

[0047] The ratio refers to the ratio of the computation time of a single layer to the data prefetching time of the next layer.

[0048] Prefetch data volume refers to the number of subsequent network layers that the prefetch engine reads in advance.

[0049] In the specific implementation, after the hardware pipeline starts parallel operation of computation and prefetching, the ratio of the computation time of a single layer to the data prefetching time of the next layer is calculated in real time. If the ratio is too high, the amount of prefetched data in subsequent layers can be increased; if the ratio is too low, the amount of prefetched data in subsequent layers can be reduced; if the ratio is within the normal range, the number of prefetching layers remains unchanged, thereby matching the prefetching speed with the computation speed and ensuring stable pipeline operation. This mechanism balances data supply and computation demand in real time, achieving continuous dynamic adaptation between the two.

[0050] In this embodiment, parameters are configured and the prefetch engine is initialized first. The first-layer model data is loaded in parallel via multiple channels of the NAND controller to start inference. During the computation phase, the computation unit and the prefetch engine work together, and the dual caches are used alternately and repeatedly to build a parallel pipeline for computation and prefetching, so as to achieve a smooth connection between each layer of computation and the data supply of the next layer. After the pipeline stabilizes, the ratio of the computation time of a single layer to the prefetch time of the next layer is calculated in real time, and the amount of prefetched data is adaptively adjusted to match the supply and computation rate, so as to stabilize the pipeline operation and improve the overall throughput efficiency of model inference.

[0051] Based on the foregoing embodiments, a second embodiment of the high-bandwidth flash memory data stream reconstruction method of the present invention is proposed. In this embodiment, step S10 may include the following sub-steps S101~S103: Sub-step S101 involves writing the model layer number, single-layer weight size, single-layer KV cache size, weight base address, and KV cache base address into the configuration register inside the bare die substrate, thus completing the loading and configuration of the model's basic hardware parameters.

[0052] In this embodiment, the HBF Base Die architecture is used to support the LLaMA-3 70B model (80-layer Transformer). In this scenario, the model weights are compressed from 140GB to 70GB after INT4 quantization, and a 40GB KV cache is configured for each 128K context. The storage system includes eight HBF stacks (512GB per stack, total capacity 4TB) and 16-layer stacked NAND flash memory (32GB per die, 16 dies integrated per stack), thus constructing a high-capacity, low-latency storage system to meet the real-time data access requirements of large model inference. Detailed hardware parameter configurations are shown in Table 1 below.

[0053] Table 1 Hardware Parameter Configuration Table

[0054] Base Die process 5nm Advanced Logic Process L1 cache capacity 16MB 8 channels × 2MB L2 cache capacity 64MB Double buffer design Prefetch queue depth 128 articles Up to 128 prefetch commands can be cached. NAND channel count 8 Number of parallel channels Number of planes per channel 4 Multi-plane parallel support Prefetch depth adjustment range Floors 1-4 Dynamic adjustment Scheduler Priority Queue Level 3 Key / Prefetch / Backend

[0055] In the specific implementation, the core parameters of the model are written into the configuration register inside the bare die of the substrate to complete the loading and configuration of the basic hardware parameters of the model. Specifically, the core parameters include five items: the number of model layers (e.g., setting the value L=80); the size of the single-layer weights after INT4 quantization (W_layer = 875MB); the size of the single-layer KV cache adapted to a 128K context length (KV_layer = 512MB); the weight base address (BASE_WEIGHT = 0x0000_0000), which serves as the physical starting address of the first-layer weight data; and the KV cache base address (BASE_KV = 0x2000_0000), which is used to store dynamic intermediate data generated during inference. After the configuration is written, it can provide accurate addressing basis for the computing unit (including its internal address generator), ensuring that subsequent data prefetching and model calculation match the model specification parameters.

[0056] Sub-step S102 performs initialization operations on the prefetch engine, including setting the initial prefetch depth, clearing all pending tasks in the prefetch queue, and synchronously marking the internal double buffer of the associated L2 cache as having no valid data, so that the prefetch engine switches to the scheduled ready state.

[0057] The initial prefetch depth refers to the number of data levels that the prefetch engine can preload in a single operation, as pre-defined.

[0058] As an example, when the computation unit processes the weights of the first layer, the prefetch engine simultaneously prefetches data from subsequent layers. When the initial prefetch depth D=1, only the immediately following layer is prefetched; when D=3, the data from the next three layers are prefetched consecutively.

[0059] The prefetch queue refers to the first-in-first-out storage structure within the prefetch engine used to temporarily store tasks to be scheduled for prefetching.

[0060] The state of no valid data means that there is no newly read weight or KV cache data after the buffer is initialized, and the old data is invalid; the computing unit and the prefetch engine do not read the data in this buffer.

[0061] The "ready to be scheduled" state means that the prefetch engine has completed all initializations, and the prefetch depth, prefetch queue, and buffer state are all configured in place, ready to receive address information and start prefetching operations at any time.

[0062] In the specific implementation, the internal control circuit of the substrate bare die issues multiple sets of initialization instructions. First, the preset initial prefetch depth is written to the prefetch engine, and then a reset signal is sent to clear all tasks to be executed in the prefetch queue. At the same time, a status marking instruction is issued to mark the double buffer inside the L2 cache as having no valid data. After all operations are completed, the hardware state machine switches the prefetch engine to the scheduled ready state, thereby ensuring that the data loading process runs according to the preset strategy from the first layer.

[0063] In sub-step S103, the storage address range of the first layer data is calculated based on the loaded weight starting base address. The prefetch engine converts the storage address range into a NAND read instruction and sends it to the NAND controller. The NAND controller then reads the original weight data from the stacked NAND core flash memory through a multi-channel parallel architecture. The original weight data is sent to the L1 cache to complete ECC error correction and decoding. The processed data is stored in the first buffer of the L2 cache and the first buffer is marked as readable and ready. The loading of the first layer data is then completed.

[0064] To support a deterministic prefetching mechanism, the model's data is linearized or stored hierarchically in the NAND core layer according to the Transformer's computation graph structure. The specific data distribution structure is shown in Table 2 below.

[0065] Table 2 Transformer Calculation Graph

[0066] Computation Operator Attention FFN Attention … FFN Data flow ↓ ↓ ↓ … ↓ Related storage data Weighted V1KV Cache Weighted V2KV Cache Weighted V3KV Cache … Weighted VLKV Cache

[0067] As shown in Table 2 above, each layer (Layer i) contains a corresponding attention mechanism or feedforward neural network (FFN) operator, and is associated with and stores the key-value cache of that layer and specific weight data (weights Vi). This structured data layout provides a physical basis for subsequent address verification.

[0068] In the specific implementation of address calculation, the address generator located inside the computing unit accurately calculates the physical address of the next layer of data based on its inference progress: when the computing unit starts the i-th layer operation, it triggers the address generator based on the known index state of the (i-1)-th layer, which outputs the physical address of the data required for the next layer (i+1-th layer) and sends the address to the prefetch engine to start the subsequent hardware-level data prefetch operation.

[0069] For specific addressing logic, the address generator uses the following two independent formulas to locate the weights and KV cache data respectively: Weight address = base address + (i+1) × single-layer weight size KV address = KV base address + (i+1) × single-layer KV size + Token offset Among them, the base address represents the starting physical address of the model weight storage area in the preset and configuration register; i represents the index of the currently being calculated level; the single-layer weight size is the storage space occupied by the number of single-layer parameters determined by the model architecture; the single-layer KV size represents the storage space occupied by the single-layer KV cache; and the token offset represents the relative position index of the currently processed token within the KV cache area of ​​this layer, which is used to accurately locate the key-value pair data of the specific token.

[0070] In the specific implementation, the address generator inside the computing unit reads the weight starting base address stored in the configuration register, and calculates the storage address range corresponding to the first-layer weight data by combining it with the single-layer weight size. Next, the prefetch engine obtains the storage address range, converts it into a read instruction adapted to the NAND hardware, and sends it to the NAND controller. Then, the NAND controller responds to the command and reads the first-layer raw weight data from the stacked NAND core flash memory through a multi-channel parallel architecture. Subsequently, the raw weight data is first sent to the L1 cache, where the L1 cache hardware circuit performs ECC error correction and data decoding. The processed first-layer weight data is stored in the first buffer of the L2 cache, and the hardware synchronously marks the first buffer as readable and ready, completing the loading of the first-layer weight data and ensuring that the first-layer model data is efficiently and error-free supplied to the computing unit.

[0071] In this embodiment, by configuring parameters, initializing the prefetch engine, and reading the first-layer weight data in parallel through multiple channels of the NAND controller, combined with L1 cache ECC error correction decoding and L2 cache state marking management, the model inference startup loading speed is accelerated, the integrity and accuracy of data reading and transmission are ensured, the timing coordination of the prefetch, cache, and computation hardware modules is stable, cache resources are used efficiently, and the startup loading requirements of large models of different specifications can be adapted.

[0072] Based on the foregoing embodiments, a third embodiment of the high-bandwidth flash memory data stream reconstruction method of the present invention is proposed. In this embodiment, step S20 may include the following sub-steps S201~S202: In sub-step S201, the pipeline startup initialization phase, the computing unit reads the first layer data of the model from the first buffer of the L2 cache and performs self-attention mechanism operation or feedforward network operation; the prefetch engine synchronously obtains the storage address of the second layer weight data and the KV cache, and after address resolution, continuous address merging and NAND physical address conversion, generates multi-plane parallel prefetch commands and adds them to the prefetch queue; the NAND controller loads the second layer data into the second buffer of the L2 cache through multi-channel parallel reading according to the prefetch commands in the prefetch queue, and marks the second buffer as ready, completing the pipeline initialization setup.

[0073] Among them, the multi-plane parallel prefetch command refers to the batch loading of weighted and KV cached data based on the NAND multi-plane architecture, relying on the parallel reading method of multiple storage planes, and combining address merging and multi-channel transmission mechanism. It can effectively improve data prefetching efficiency and is the core instruction for building a high-speed hardware prefetch pipeline.

[0074] Please see Figure 3 The diagram illustrates the L2 cache double buffer structure: The L2 cache adopts a double buffer design, with buffer A (i.e., the first buffer) and buffer B (i.e., the second buffer) both being 32MB, respectively caching adjacent levels of data; when the calculation of the i-th layer is completed and the data of the i+1-th layer is ready, the switching control logic switches the data reading object of the calculation unit, and the prefetch engine synchronously writes the data of the i+2-th layer to the idle buffer.

[0075] Please refer to the figure. Figure 4 The diagram illustrates the stages of the data processing pipeline. The pipeline is divided into four stages: NAND prefetch reading, hardware ECC decoding, quantization format conversion, and computing unit consumption. Each stage is executed in parallel. The pipeline relies on a dual-buffered switching and multi-stage pipeline parallel mechanism to ensure efficient throughput of AI inference data streams.

[0076] In the specific implementation, during the pipeline startup initialization phase, the computing unit reads the first-layer weight data of the model from the first buffer of the L2 cache and performs self-attention calculation or feedforward network calculation. Simultaneously, the prefetch engine synchronously obtains the storage addresses of the second-layer weights and the KV cache, and generates multi-plane parallel prefetch commands through address resolution, contiguous address merging, and NAND physical address translation, adding the commands to the prefetch queue. Then, the NAND controller, based on the prefetch commands in the queue, reads the second-layer data in parallel through multiple channels, loads it into the second buffer of the L2 cache, and marks the buffer as ready, completing the pipeline initialization setup. The address merging and multi-plane, multi-channel parallel mechanisms effectively improve prefetch throughput efficiency.

[0077] In sub-step S202, after the first-layer computation task is completed and the data in the second buffer is ready, the data reading path of the computation unit is switched to the second buffer, and the second-layer computation of the model is started. Simultaneously, the storage address of the third-layer data is obtained by the prefetch engine. The address optimization and parallel prefetching process are repeated to prefetch the third-layer data into the empty first buffer. In the computation process of subsequent layers of the model, this process continues to iterate. Relying on the working mode of alternating switching and cyclic reuse of dual buffers, the computation of each layer of the model and the prefetching of the next layer of data are completely overlapped in the time dimension, continuously realizing the continuous parallel execution of computation and data preparation, forming a stable hardware prefetch pipeline.

[0078] In the specific implementation, after the first-layer computation task is completed and the second-layer data prefetched in the second buffer is ready, the data reading path of the computation unit is immediately switched from reading the first buffer to reading the second buffer, and the second-layer Transformer computation task of the model is officially started. At the same time, the prefetch engine synchronously triggers parallel prefetching operations to obtain the storage addresses of the model's third-layer weights and KV cache data in real time, and repeatedly executes preprocessing processes such as address resolution, continuous address merging optimization, and NAND physical address conversion to generate multi-plane parallel prefetch commands for the third-layer data. Through multi-channel parallel reading of the NAND controller, the third-layer data is prefetched into the first buffer that has completed the first-layer computation and is in an empty state, and the ready state flag of the first buffer is updated. The inference computation of all subsequent layers of the model follows this logic and continues to iterate: the currently ready buffer always provides the computation unit with layer computation data, while the prefetch engine uses another buffer that has just been released to complete the background prefetch loading of the next layer of model data. Based on the core working mechanism of alternating switching and cyclic reuse of L2 cache dual buffers, the forward computation process of each layer of the model can completely overlap with the data prefetching preparation process of the next layer in the time dimension, completely eliminating the data loading waiting latency, continuously maintaining the parallel operation of computation tasks and data prefetching tasks, and finally building a continuous, stable and efficient hardware pipeline to achieve seamless connection and efficient iteration of multi-level inference of the model.

[0079] Furthermore, in one embodiment, in order to improve prefetch efficiency and reduce bus bandwidth usage, a continuous address verification and physical address mapping mechanism can be introduced during the address resolution and merging optimization process (i.e., in the specific implementation of sub-step S201), specifically including the following sub-steps A10~A30: Sub-step A10 involves the built-in address generator of the computing unit retrieving the starting base address of the second-layer weight and the starting base address of the KV cache, integrating the two types of base addresses to calculate the corresponding storage address of the second layer, and sending it to the prefetch engine. Sub-step A20 involves the prefetch engine parsing the second-level storage address and comparing it one by one with the existing pending addresses in the prefetch queue to verify whether the parsed storage address meets the merging condition. If the condition is met, the consecutive and overlapping addresses are merged to complete the merging; otherwise, the parsed storage address is retained separately. Sub-step A30: After completing address merging, discrimination and processing, extract the valid logical address and convert it into a physical address that is compatible with the NAND controller; organize the prefetch tasks according to the physical address arrangement order, generate multi-plane parallel prefetch commands accordingly, store them in the prefetch queue and update the queue task list synchronously.

[0080] In its implementation, the computation unit's built-in address generator retrieves the base address of the second-layer weights and KV cache. After integration and calculation, the storage address of the second layer is obtained and sent to the prefetch engine. The prefetch engine then parses this address and compares it with each existing requested address in the prefetch queue. If the merging conditions are met, consecutive or overlapping addresses are merged; otherwise, they are retained as independent entries. Subsequently, the prefetch engine extracts the valid logical address and converts it into a physical address adapted to the NAND controller. Based on this, prefetch tasks are organized according to the physical address arrangement, generating multi-plane parallel prefetch commands, which are stored in the prefetch queue and the task list is updated synchronously. This mechanism, through intelligent address merging and multi-plane parallel prefetching, effectively reduces bus bandwidth usage and instruction overhead, fully releasing NAND storage throughput, and thus achieving efficient pipelined parallelism of computation and storage.

[0081] Furthermore, in one embodiment, in order to fully utilize the concurrency characteristics of the NAND Core Stack and improve data loading speed, a multi-plane parallel scheduling and state synchronization mechanism can be introduced during the execution of multi-channel data reading and cache writing (i.e., in the specific implementation of sub-step S201), specifically including the following sub-steps B10~B20: Sub-step B10 involves the NAND controller reading and parsing the multi-plane parallel prefetch commands in the prefetch queue, extracting the target physical address and data length information, and distributing the read tasks to the corresponding NAND channels and logic cells. In sub-step B20, the NAND controller controls each NAND channel to synchronously read the second layer weight and KV cache data according to the assigned read task, and writes the read data into the second buffer of the L2 cache; after the data writing is completed, the second buffer is marked as ready.

[0082] In the specific implementation, the NAND controller reads and parses the multi-plane parallel prefetch instructions in the prefetch queue, extracts the target physical address and data length, and distributes the read tasks to the corresponding NAND channels and logic cells. Then, the NAND controller schedules each NAND channel to synchronously read the second-layer weights and KV cache data according to the assigned read tasks, and writes the read data into the second buffer of the L2 cache. After all data is written, the status of the second buffer is marked as ready. Thus, multi-channel, multi-plane parallel reading accelerates the loading speed of lower-layer data, ensures the integrity of cached data, eliminates computational waiting latency, and improves the overall inference throughput performance.

[0083] In this embodiment, data prefetching is driven by the computation progress at the model level. Relying on the dual-buffer alternating reuse architecture, computation at each layer and data loading at the next layer are carried out in parallel. Combined with address merging and multi-plane multi-channel high-speed prefetching mechanism, idle latency while waiting for data is eliminated, and the inference pipeline is continuously maintained at full load, thereby improving the overall inference throughput rate.

[0084] Based on the foregoing embodiments, a fourth embodiment of the high-bandwidth flash memory data stream reconstruction method of the present invention is proposed. In this embodiment, step S30 may include the following sub-steps S301~S303: Sub-step S301: When the ratio of the computation time of a single layer to the data prefetching time of the next layer exceeds a preset high threshold, increase the prefetching depth and load more subsequent layer data in advance.

[0085] Among them, the preset high threshold refers to the preset judgment value; the preferred value is 1.2, which means that the calculation time of a single layer is 1.2 times the calculation time of the next layer.

[0086] In practice, when the ratio of the computation time of a single layer to the data prefetching time of the next layer is too large, the prefetching depth of subsequent layers can be increased. The higher the prefetching depth, the more subsequent network layers the prefetching engine can extract and read, and the total amount of prefetched data will be increased accordingly.

[0087] Sub-step S302: When the ratio of the computation time of a single layer to the data prefetching time of the next layer is lower than a preset low threshold, reduce the prefetching depth to reduce unnecessary data overhead.

[0088] Among them, the preset low threshold refers to another preset judgment value; the preferred value is 0.8, which means that the calculation time of a single layer is 0.8 times the calculation time of the next layer.

[0089] In practice, when the ratio is small, the prefetch depth of subsequent layers can be reduced; the lower the prefetch depth, the fewer subsequent network layers the prefetch engine reads in advance, and the total amount of prefetched data is reduced accordingly.

[0090] Sub-step S303: When the ratio of the computation time of a single layer to the data prefetching time of the next layer is within a reasonable range, the current prefetching depth is maintained unchanged; wherein, the reasonable range is between a preset high threshold and a preset low threshold.

[0091] The reasonable range refers to the numerical range between the preset low threshold of 0.8 and the preset high threshold of 1.2.

[0092] In the implementation, when the ratio is within a reasonable range, the current prefetch depth remains unchanged; the number of subsequent network layers read by the prefetch engine and the total amount of prefetched data remain stable. This adaptive adjustment mechanism matches the prefetch speed with the computation speed, ensuring stable operation of the hardware pipeline.

[0093] In addition, multi-level prefetching tasks are set with high and low priorities. Data immediately adjacent to the next level is read first, while data from more distant levels is read using idle bandwidth, thus avoiding contention for core bandwidth.

[0094] In this embodiment, an adaptive adjustment mechanism is used to adapt to the fluctuating read latency of NAND flash memory, which can balance the running speed of computation and prefetching, reduce bandwidth idleness and pipeline lag, and simultaneously take into account device performance and operational stability.

[0095] Based on the foregoing embodiments, a fifth embodiment of the high-bandwidth flash memory data stream reconstruction method of the present invention is proposed. In this embodiment, after step S20, the method may further include the following steps S40 to S70: In step S40, the prefetch engine continuously monitors the prefetching time of data at each layer. When the prefetching time exceeds the preset time threshold, a prefetching anomaly is determined, and an anomaly signal is reported to the multi-plane scheduler built into the NAND controller, and the anomaly handling mechanism is started synchronously.

[0096] The preset time threshold refers to the pre-configured maximum allowable time for prefetching a single-level NAND (e.g., 100μs).

[0097] An abnormal signal refers to a hardware status notification signal generated and reported by the prefetch engine after it detects that the prefetch time of a single-level NAND exceeds a preset time threshold (100μs).

[0098] The exception handling mechanism refers to the hardware fault tolerance and control mechanism that accompanies the NAND prefetch pipeline. It is triggered by the prefetch engine and executed collaboratively by multiple plane schedulers. The core includes the entire process logic of exception reporting, task priority adjustment, non-urgent task suspension, computing unit fault tolerance waiting, pipeline parallel fallback, and post-event adaptive parameter tuning. It is used to solve the prefetch timeout problem caused by NAND read latency exceptions.

[0099] In the specific implementation, the prefetch engine continuously collects the prefetch time of data from each layer and compares the real-time time of each layer with the preset time threshold in real time. When the prefetch time of a single instance exceeds the preset time threshold, the prefetch engine determines that the prefetch task of the current layer has a timeout exception and locks the prefetch task information corresponding to the timeout layer. Subsequently, the prefetch engine generates an exception signal carrying the timeout layer information and pushes the signal to the multi-plane scheduler built into the NAND controller, which locally triggers the prefetch exception handling mechanism and executes the subsequent resource scheduling and fault tolerance processing process.

[0100] In step S50, after receiving the abnormal signal, the multi-plane scheduler sets the prefetch command corresponding to the timeout layer to the priority scheduling level, prioritizes the scheduling of computing and storage resources to perform data loading, and simultaneously suspends various non-urgent background tasks to concentrate resources to ensure that the data of this layer is loaded in a timely manner.

[0101] Among them, the priority scheduling level refers to the higher resource scheduling authority level set by the multi-plane scheduler for timeout layer prefetch commands.

[0102] Non-urgent background tasks refer to secondary background tasks that do not affect the current level of core computing and critical data loading during the NAND prefetch pipeline operation. Specifically, they include two types of tasks: deep prefetch and garbage collection.

[0103] In the specific implementation, after receiving a timeout exception signal reported by the prefetch engine, the multi-plane scheduler refreshes the scheduling priority of the timeout layer prefetch commands, forcibly mapping them to a higher scheduling level. This allows them to prioritize acquiring computing and storage resources in the task queue and preempt hardware scheduling time. Simultaneously, the multi-plane scheduler suspends various non-urgent background tasks, reclaims redundant system resources, and centrally supplies them for timeout layer data loading. Before the timeout layer data loading is complete, the scheduling policy and resource allocation status are continuously locked, maintaining the high priority of timeout layer prefetch commands and the suspension of non-urgent background tasks until the prefetch job at this layer is completed, at which point the pipeline exits the exception handling state.

[0104] In step S60, if the current level of computation is about to be completed, but the prefetched data it depends on is not ready and the computation unit is about to enter a waiting state, then the operation of the computation unit is paused and a timeout waiting counter is started to record the waiting time. During this period, the hardware pipeline relies on the dual-buffer alternating scheduling mechanism to schedule other data-ready levels to execute computation tasks, maintaining the basic parallel operation of the pipeline.

[0105] In the specific implementation, during the prefetch timeout exception handling process, the status of the current level of computation tasks is monitored in real time. If the current level of computation task is about to complete, but its dependent prefetched data is not ready and the computation unit is about to enter a waiting state, the operation of that computation unit is immediately suspended, and a timeout wait counter is started to record the waiting time, preventing the computation unit from running idly and the pipeline from becoming abnormal. During the blocking wait period at this level, the hardware pipeline uses a built-in dual-buffer alternating scheduling mechanism to isolate the unready blocking level and schedule other data-ready levels to execute computation tasks. By alternating caching with dual buffers and switching the computation context and intermediate data of each level, seamless task relay scheduling is achieved, avoiding pipeline stagnation caused by single-level prefetch blocking, continuously maintaining the pipeline's parallel operation capability, and improving hardware resource utilization.

[0106] Step S70: After the timeout layer data is loaded, the computing unit resumes operation immediately, and the prefetch depth and resource allocation weight of subsequent layers are dynamically adjusted according to the timeout record. If the prefetch timeout is triggered multiple times in subsequent layers, the prefetch depth is reduced level by level. If the prefetch timeout frequency of subsequent layers continues to decrease, the prefetch depth is increased level by level.

[0107] In the specific implementation, after the timeout layer data is loaded, the running state of the suspended computing unit is immediately restored. Based on the timeout record data, the prefetch depth and resource allocation weight of subsequent layers are dynamically adjusted to complete the single-anomaly closed-loop optimization. Subsequently, adaptive adjustment is achieved by relying on the timeout frequency statistics mechanism: if the prefetch timeout is triggered multiple times in subsequent layers, it is determined that the prefetch lead is too high, and the prefetch depth of each layer is reduced step by step to reduce the risk of pipeline blockage; if the frequency of subsequent prefetch timeouts continues to decrease, it is determined that the prefetch margin is insufficient, and the prefetch depth is increased step by step to give full play to the prefetch acceleration advantage and dynamically balance the data prefetch efficiency and operational stability of the pipeline.

[0108] In this embodiment, through the above-mentioned exception handling mechanism, system resources can be dynamically allocated and prefetching strategies can be adaptively optimized in prefetching timeout scenarios, avoiding interruption of computing tasks and pipeline stalls, and effectively balancing the predictability of data prefetching with overall operational stability.

[0109] Based on the same inventive concept, the sixth embodiment of this invention also provides a high-bandwidth flash memory data stream reconstruction system corresponding to the high-bandwidth flash memory data stream reconstruction method of the foregoing embodiments. Since the principle of the system in the sixth embodiment of this invention for solving the problem is similar to the high-bandwidth flash memory data stream reconstruction method of the foregoing embodiments of this invention, the implementation of the system can refer to the implementation of the method, and repeated details will not be elaborated further. Please refer to... Figure 5 This invention relates to a high-bandwidth flash memory data stream reconstruction system, applied to an HBF chip comprising a substrate die and NAND flash memory, wherein the substrate die integrates a computing unit, a prefetch engine, a NAND controller, and an L2 cache with dual cache areas. The system may include: Initialization module 10 is used to configure model hardware parameters, initialize the prefetch engine, and control the NAND controller to read the first layer data of the model in parallel through multiple channels to complete the startup loading of model inference calculation; Module 20 is used for model computation. Based on the parallel collaboration between the computation unit and the prefetch engine, and by utilizing the alternating switching and circular reuse mechanism of the double buffer in the L2 cache, a hardware pipeline for parallel execution of computation and data prefetching is constructed to achieve a smooth connection between the computation of each layer of the model and the data supply of the next layer. The adjustment module 30 is used to continuously calculate the ratio of the computation time of a single layer to the data prefetching time of the next layer after the pipeline enters a stable state, and dynamically adjust the amount of prefetched data in subsequent layers based on the ratio to match the data supply speed with the computation speed, so as to ensure the stable operation of the pipeline.

[0110] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described high-bandwidth flash data stream reconstruction method.

[0111] Figure 6 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 6 The general will label all buses as bus systems.

[0112] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0113] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0114] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic device 400. Examples of this data include: any executable program for operation on the electronic device 400, such as the operating system 4021 and application programs 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The high-bandwidth flash data stream reconstruction method provided in this embodiment of the invention can be included in the application program 4022.

[0115] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the high-bandwidth flash memory data stream reconstruction method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0116] In an exemplary embodiment, the electronic device 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0117] In summary, this invention first configures parameters and initializes the prefetch engine, then loads the first-layer model data in parallel via multiple channels of the NAND controller to initiate inference. During the computation phase, the computation unit and the prefetch engine work together, and a parallel computation prefetch pipeline is built by alternating and reusing dual caches, achieving a smooth connection between each layer's computation and the data supply to the next layer. After the pipeline stabilizes, the ratio of single-layer computation to the prefetch time of the next layer is statistically analyzed in real time, and the amount of prefetched data is adaptively adjusted to match the supply and computation rate, thus stabilizing pipeline operation and improving the overall throughput efficiency of model inference.

[0118] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for reconstructing a high-bandwidth flash memory data stream, characterized in that, An HBF chip comprising a substrate die and NAND flash memory, wherein the substrate die integrates a computing unit, a prefetch engine, a NAND controller, and an L2 cache with dual caches, the method comprising: Configure the model hardware parameters, initialize the prefetch engine, and control the NAND controller to read the first layer data of the model in parallel through multiple channels to complete the startup loading of model inference computation; During model computation, a hardware pipeline for parallel execution of computation and data prefetching is constructed based on the parallel collaboration between the computation unit and the prefetching engine, and by utilizing the alternating switching and circular reuse mechanism of the double buffers in the L2 cache, so as to achieve a smooth connection between the computation of each layer of the model and the data supply of the lower layer. Once the pipeline reaches a stable state, the ratio of the computation time of a single layer to the data prefetching time of the next layer is continuously calculated. Based on this ratio, the amount of prefetched data for subsequent layers is dynamically adjusted to match the data supply speed with the computation speed, ensuring stable pipeline operation.

2. The method according to claim 1, characterized in that, The substrate bare die also integrates an L1 cache. The configuration of model hardware parameters, initialization of the prefetch engine, and control of the NAND controller to read the model's first-layer data in parallel through multiple channels complete the startup loading of model inference computation, including: Write the model layer number, single-layer weight size, single-layer KV cache size, weight starting base address, and KV cache starting base address into the configuration register inside the bare die substrate to complete the loading and configuration of the model's basic hardware parameters; The prefetch engine is initialized, including setting the initial prefetch depth, clearing all pending tasks in the prefetch queue, and simultaneously marking the internal double buffers of the associated L2 cache as having no valid data, so that the prefetch engine is switched to the scheduling ready state. Based on the loaded weight starting base address, the storage address range of the first layer data is calculated. The prefetch engine converts the storage address range into NAND read instructions and sends them to the NAND controller. The NAND controller then reads the original weight data from the stacked NAND core flash memory through a multi-channel parallel architecture. The original weight data is sent to the L1 cache to complete ECC error correction and decoding. The processed data is stored in the first buffer of the L2 cache and marked as readable and ready. The loading of the first layer data is then completed.

3. The method according to claim 1, characterized in that, The L2 cache includes a first buffer and a second buffer. During model computation, based on the parallel collaboration between the computation unit and the prefetch engine, and utilizing the alternating switching and circular reuse mechanism of the two buffers within the L2 cache, a hardware pipeline for parallel execution of computation and data prefetching is constructed to achieve a smooth connection between the computation of each layer of the model and the data supply of the next layer, including: During the pipeline startup initialization phase, the computing unit reads the first layer data of the model from the first buffer of the L2 cache and performs self-attention mechanism operations or feedforward network operations. The prefetch engine synchronously obtains the storage address of the second layer weight data and the KV cache. After address resolution, contiguous address merging, and NAND physical address translation, it generates multi-plane parallel prefetch commands and adds them to the prefetch queue. The NAND controller loads the second layer data into the second buffer of the L2 cache through multi-channel parallel reading according to the prefetch commands in the prefetch queue, and marks the second buffer as ready, completing the pipeline initialization setup. Once the first-layer computation task is completed and the data in the second buffer is ready, the data reading path of the computation unit is switched to the second buffer, and the second-layer computation of the model is started. Simultaneously, the storage address of the third-layer data is obtained by the prefetch engine. Through address optimization and parallel prefetching processes, the third-layer data is prefetched into the vacant first buffer. This process continues to iterate in subsequent layers of the model. Relying on the working mode of alternating switching and cyclic reuse of dual buffers, the computation of each layer of the model and the prefetching of the next layer of data are completely overlapped in the time dimension, continuously realizing the continuous parallel execution of computation and data preparation, forming a stable hardware pipeline.

4. The method according to claim 3, characterized in that, The process involves the prefetch engine synchronously acquiring the storage addresses of the second-layer weight data and the KV cache, performing address resolution, contiguous address merging, and NAND physical address translation, and then generating multi-plane parallel prefetch commands and adding them to the prefetch queue. This includes: The built-in address generator of the computing unit retrieves the starting base address of the second layer weight and the starting base address of the KV cache, integrates the two types of base addresses to calculate the corresponding storage address of the second layer, and sends it to the prefetch engine. The prefetch engine parses the storage address of the second layer and compares it with the existing pending addresses in the prefetch queue one by one to verify whether the parsed storage address meets the merging condition. If the condition is met, the consecutive and overlapping addresses are merged to complete the merging; if not, the parsed storage address is retained separately. After address merging, discrimination, and processing are completed, the valid logical addresses are extracted and converted into physical addresses that are compatible with the NAND controller. The prefetch tasks are organized according to the physical address arrangement order, and multi-plane parallel prefetch commands are generated accordingly. These commands are stored in the prefetch queue and the queue task list is updated synchronously.

5. The method according to claim 3, characterized in that, The process of loading the second layer data into the second buffer of the L2 cache by the NAND controller according to the prefetch commands in the prefetch queue through a multi-channel parallel reading method, and marking the second buffer as ready, includes: The NAND controller reads and parses the multi-plane parallel prefetch commands in the prefetch queue, extracts the target physical address and data length information, and distributes the read tasks to the corresponding NAND channels and logic cells. The NAND controller controls each NAND channel to synchronously read the second layer weight and KV cache data according to the assigned read task, and writes the read data into the second buffer of the L2 cache; after the data writing is completed, the second buffer is marked as ready.

6. The method according to claim 1, characterized in that, Following the steps of constructing a hardware pipeline for parallel execution of computation and data prefetching based on the parallel collaboration of the computing unit and the prefetching engine, and utilizing the alternating switching and circular reuse mechanism of the double buffers within the L2 cache to achieve a smooth connection between the computation of each layer of the model and the data supply of the next layer, the following is also included: The prefetch engine continuously monitors the prefetching time of data at each layer. When the prefetching time exceeds the preset time threshold, a prefetching anomaly is determined, and an anomaly signal is reported to the multi-plane scheduler built into the NAND controller, and the anomaly handling mechanism is started synchronously. After receiving an anomaly signal, the multi-plane scheduler sets the prefetch command corresponding to the timeout layer as the priority scheduling level, prioritizes the scheduling of computing and storage resources to perform data loading; and simultaneously suspends various non-urgent background tasks to concentrate resources to ensure that the data at this layer is loaded and ready in a timely manner. If the current level of computation is about to complete, but the prefetched data it depends on is not ready and the computation unit is about to enter a waiting state, the operation of the computation unit is paused and a timeout counter is started to record the waiting time. During this period, the hardware pipeline relies on the dual-buffer alternating scheduling mechanism to schedule other data-ready levels to execute computation tasks, maintaining the basic parallel operation of the pipeline. Once the timeout layer data is loaded, the computing unit resumes operation immediately, and the prefetch depth and resource allocation weight of subsequent layers are dynamically adjusted based on the timeout record. If a subsequent layer triggers a prefetch timeout multiple times, the prefetch depth is reduced level by level. If the frequency of prefetch timeouts in subsequent layers continues to decrease, the prefetch depth is increased level by level.

7. The method according to claim 1, characterized in that, The dynamic adjustment of the prefetch data volume of subsequent levels based on this ratio includes: When the ratio of the computation time of a single layer to the data prefetching time of the next layer exceeds the preset high threshold, the prefetching depth is increased to load more subsequent layer data in advance. When the ratio of the computation time of a single layer to the data prefetching time of the next layer is lower than a preset low threshold, the prefetching depth is reduced to cut unnecessary data overhead. When the ratio of the computation time of a single layer to the data prefetching time of the next layer is within a reasonable range, the current prefetching depth remains unchanged; wherein, the reasonable range is between a preset high threshold and a preset low threshold.

8. A high-bandwidth flash memory data stream reconstruction system, characterized in that, An HBF chip, comprising a substrate die and NAND flash memory, wherein the substrate die integrates a computing unit, a prefetch engine, a NAND controller, and an L2 cache with dual caches, is used in the system. The initialization module is used to configure the model hardware parameters, initialize the prefetch engine, and control the NAND controller to read the first layer data of the model in parallel through multiple channels, thus completing the startup loading of the model inference computation. The building module is used for model computation. Based on the parallel collaboration between the computing unit and the prefetching engine, and utilizing the alternating switching and circular reuse mechanism of the double buffer in the L2 cache, a hardware pipeline for parallel execution of computation and data prefetching is constructed to achieve a smooth connection between the computation of each layer of the model and the data supply of the lower layer. The adjustment module is used to continuously calculate the ratio of the computation time of a single layer to the data prefetching time of the next layer after the pipeline enters a stable state, and dynamically adjust the amount of prefetched data in subsequent layers based on this ratio to match the data supply speed with the computation speed, ensuring stable pipeline operation.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images