UFS NAND-based in-memory computing acceleration systems and electronic devices
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI GUANGYU XINCHEN TECHNOLOGY CO LTD
- Filing Date
- 2026-02-11
- Publication Date
- 2026-05-29
Smart Images
Figure CN121680754B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of chip technology, and in particular to an in-memory computing acceleration system and electronic device based on UFS NAND. Background Technology
[0002] When deploying large models on the edge (mobile phones, edge computing devices), the sheer volume of model parameters, reaching hundreds of billions to trillions, means that the DRAM (Dynamic Random Access Memory) on the edge (typically between 4GB and 16GB) and on-chip cache are far from sufficient to accommodate all the parameters. Therefore, the industry commonly uses UFS NAND (Universal Flash Storage NAND Flash Memory, a non-flash memory using the Universal Flash Storage (UFS) interface protocol) (capacity ranging from 256GB to 2TB) as the core storage medium to store the weight parameters, vocabulary, and intermediate computation templates of large models. However, existing UFS NAND only has data storage capabilities and lacks the computational power to handle large models. When performing calculations on large models (such as Transformer attention mechanisms and linear transformations), the parameters in the NAND (NAND Flash Memory) need to be frequently transferred to high-performance chips (such as APs and NPUs) on the edge via a bus. Moreover, the migration process needs to adapt to the characteristics of large models, such as "parameter access sparsity" (e.g., only some attention heads are activated during LLM inference) and "computation intensity" (e.g., QKV matrix multiplication accounts for more than 50%). This makes the existing migration schemes extremely inefficient and severely restricts the running efficiency of large models on the edge.
[0003] While in-memory computation (PIM) technology can reduce data migration to some extent, most existing solutions focus on small models (such as CNN image classification) and are not optimized for the characteristics of large models. On the one hand, these solutions lack attention computation acceleration units, making it impossible to accelerate the core attention mechanism of large models at the hardware level; for example, they cannot efficiently implement QKV matrix multiplication and Softmax operations. On the other hand, they do not adapt to the characteristics of sparse parameter access, still performing full migration for sparse parameters, wasting valuable bandwidth resources. Furthermore, the computing power of existing solutions cannot meet the requirements of large model inference; large model inference requires at least 500 GOPS FP16 computing power, which current solutions struggle to achieve. More importantly, these solutions are not designed based on the interface protocol and storage characteristics of UFS NAND, resulting in extremely poor compatibility and making it difficult to play their due role in practical applications of large models.
[0004] Existing technologies suffer from cost and compatibility issues in several aspects, severely hindering the industrialization of large models on the edge. First, the parameter migration cost for large models is extremely high. A single inference attempt requires accessing hundreds of MB to several GB of parameters (e.g., approximately 1.2 GB for a single 7B LLM inference attempt). Since existing UFS NAND lacks local computing capabilities, parameter migration takes 5ms to 20ms, accounting for over 60% of the total inference time, and consuming over 70% of the power. This results in high latency and shortened battery life for large models on the edge. Second, existing NAND logic chip computing units only support basic multiply-accumulate operations, failing to meet the core computing requirements of large models. For example, they lack attention mechanism acceleration units and do not support sparse parameter access, thus failing to cover scenarios such as LLM inference and Transformer image generation. Furthermore, the computing power of existing logic chips is mostly below 300 GOPS (FP16), while 7B LLM inference requires at least 500 GOPS, necessitating a full migration of large models to external chips, thus failing to leverage the advantages of local storage. Finally, the existing PIM solution is not compatible with the UFS 3.0 / 4.0 protocol and cannot directly replace the existing UFS NAND on the device side. It requires modification of the device motherboard design, resulting in high costs for industrialization. Summary of the Invention
[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an in-memory computing acceleration system and electronic device based on UFS NAND, which solves the technical problems of existing edge-side large model deployment technology, such as high parameter migration costs, lack of computing characteristic adaptation, and difficulty in industrialization due to the lack of local computing capabilities of UFS NAND, the in-memory computing scheme not being adapted to the characteristics of large models and insufficient computing power, and poor protocol compatibility.
[0006] To achieve the above and other related objectives, this invention provides an in-memory computing acceleration system based on UFS NAND, comprising: a UFS NAND storage array module for partitioning and storing large model parameters, parameter sparsity indexes, and intermediate data generated during large model computation; and a logic chip integration module connected to the UFS NAND storage array module for evaluating the task's computing power requirements and parameter sparsity upon receiving a request to initiate a large model task, obtaining corresponding data from the UFS NAND storage array module based on the evaluation results, performing attention-accelerated computation on the locally deployed large model to obtain output computation results, or migrating the computation to an external chip and writing the computation results back.
[0007] In one embodiment of the present invention, the UFS NAND storage array module is divided into a large model parameter area, a parameter sparse index area, and an intermediate result cache area; wherein, the large model parameter area is used to divide the large model into sub-partitions according to each attention head, so as to store the weight parameters of the corresponding attention head respectively; the parameter sparse index area is used to record the address of the effective parameters when the parameters are sparsely accessed; and the intermediate result cache area is used to temporarily store the intermediate data generated during the calculation of the large model.
[0008] In one embodiment of the present invention, the logic chip integration module includes: a UFS protocol compatible control unit for managing address mapping, bad blocks, and data verification; a large model task scheduling unit for evaluating the task's computing power requirements and parameter sparsity upon receiving a request to initiate a large model task, and generating corresponding local computing instructions or performing migration computing operations based on the evaluation results; a large model-adaptive AI computing unit for reading valid parameters located through sparse indexes upon receiving local computing instructions, using the large model deployed locally for attention-accelerated computing, and obtaining computing results; and a high-speed data buffer unit for caching data read from the UFS NAND storage array module and intermediate results generated by the large model-adaptive AI computing unit.
[0009] In one embodiment of the present invention, the UFS protocol compatible control unit supports UFS 3.0 and / or 4.0 protocols, and is further provided with a sparse parameter index subunit, which locates the address of the valid parameter of the sparse parameter index area through hardware.
[0010] In one embodiment of the present invention, the large model-adaptive AI computing unit includes: a highly parallel MAC array for providing large model multiplication and addition capabilities to adapt to linear transformations of large models; an attention computing acceleration subunit for implementing the core operations of the large model attention mechanism using hardware; wherein the core operations include: QKV matrix multiplication, Scaled Dot-Product, and Softmax operations; and a sparse data processing subunit for performing zero-value skipping processing on the sparse data in the parameter region of the large model according to the address of the valid parameters located by the sparse index, and only reading the valid parameters.
[0011] In one embodiment of the present invention, the large model task scheduling unit is used to evaluate the computing power requirements and parameter sparsity of the current large model task; if it is a task with high sparsity and light computing power, a local computing instruction is generated; if it is a task with low sparsity and dense computing power, a migration computing operation is performed.
[0012] In one embodiment of the present invention, the migration calculation operation includes: compressing the relevant parameters using the ZSTD algorithm, migrating them to an external chip for processing via a high-speed UFS interface, so that the external chip can write back the calculation results or the required intermediate data to local storage.
[0013] In one embodiment of the present invention, the UFS NAND storage array module is connected to the logic chip through 64 high-speed vertical interconnect channels.
[0014] In one embodiment of the present invention, the high-speed data buffer unit employs 8MB 8-port RAM.
[0015] To achieve the above and other related objectives, the present invention provides an electronic device that utilizes the UFS NAND-based in-memory computing acceleration system described above.
[0016] As described above, this invention is an in-memory computing acceleration system and electronic device based on UFS NAND, which has the following beneficial effects: This invention uses a UFS NAND storage array module to partition and store large model parameters, parameter sparsity indexes, and intermediate data generated during large model computation. When a request to initiate a large model task is received, the logic chip integration module evaluates the task's computing power requirements and parameter sparsity. Based on the evaluation results, it retrieves the corresponding data from the UFS NAND storage array module and uses the locally deployed large model for attention-accelerated computation to obtain the output computation result, or migrates to an external chip for computation and writes the computation result back. This invention is highly adaptable to the core characteristics of large models, supports parameter sparse access and hardware-accelerated attention mechanisms, and can meet the computational needs of large models such as LLM and Transformer. Furthermore, it enhances the computing power of the UFS NAND logic chip, is compatible with the UFS protocol and existing architecture, can directly replace edge devices without hardware modifications, and reduces costs. It can also optimize task scheduling, intelligently switching between local computation and external migration based on task type and parameter sparsity, balancing efficiency and performance. Attached Figure Description
[0017] Figure 1 The diagram shown is a schematic representation of an in-memory computing acceleration system based on UFS NAND according to an embodiment of the present invention.
[0018] Figure 2 The diagram shown is a schematic representation of an in-memory computing acceleration system based on UFS NAND according to an embodiment of the present invention.
[0019] Figure 3 The diagram shown is a structural schematic of a large model-adaptive AI computing unit according to an embodiment of the present invention. Detailed Implementation
[0020] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.
[0021] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of the invention. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of the invention. The following detailed description should not be considered limiting, and the scope of the embodiments of the invention is defined only by the claims of the published patents. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used herein to illustrate the relationship between one element or feature shown in the figures and another element or feature.
[0022] Throughout this specification, when it is said that a part is "connected" to another part, this includes not only "direct connection" but also "indirect connection" by placing other elements in between. Furthermore, when it is said that a part "includes" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather means that other constituent elements may also be included.
[0023] The terms "first," "second," and "third," etc., used herein are for the purpose of describing various parts, components, regions, layers, and / or segments, but are not limiting. These terms are used only to distinguish one part, component, region, layer, or segment from others. Therefore, the "first part," "component," "region," "layer," or "segment" described below may refer to a "second part," "component," "region," "layer," or "segment" without departing from the scope of this invention.
[0024] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.
[0025] This invention provides an in-memory computing acceleration system based on UFS NAND. It utilizes a UFS NAND storage array module to partition and store large model parameters, parameter sparsity indexes, and intermediate data generated during large model computation. Upon receiving a request to initiate a large model task, the logic chip integration module assesses the task's computing power requirements and parameter sparsity. Based on the assessment results, it retrieves the corresponding data from the UFS NAND storage array module and performs attention-accelerated computation on the locally deployed large model to obtain the output computation result, or migrates the computation to an external chip and writes the result back. This invention is highly adaptable to the core characteristics of large models, supports parameter sparse access and hardware-accelerated attention mechanisms, and can meet the computational needs of large models such as LLM and Transformer. Furthermore, it enhances the computing power of the UFS NAND logic chip, is compatible with the UFS protocol and existing architectures, and can directly replace edge devices without hardware modifications, reducing costs. It also optimizes task scheduling, intelligently switching between local computation and external migration based on task type and parameter sparsity, balancing efficiency and performance.
[0026] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:
[0027] <1> UFS (Universal Flash Storage) is a type of universal flash storage.
[0028] <2> DRAM (Dynamic Random Access Memory)
[0029] <3> LLM (Large Language Model)
[0030] <4> NAND (NAND Flash Memory), as opposed to non-flash memory;
[0031] <5> AP (Application Processor);
[0032] <6> NPU (Neural Processing Unit) is a neural network processor.
[0033] <7> PIM (Processing-In-Memory) refers to in-memory computation.
[0034] <8> QKV (Query-Key-Value) is a query-key-value matrix.
[0035] <9> GOPS (Giga Operations Per Second) means one billion operations per second.
[0036] <10> RAM (Random Access Memory)
[0037] <11> CNN (Convolutional Neural Network)
[0038] <12> 3D TLC UFS 4.0 NAND (3D Triple-Level Cell Universal Flash Storage 4.0 NAND), 3D triple-level cell universal flash storage 4.0 flash memory;
[0039] <13> LDPC ECC (Low-Density Parity-Check Error Correction Code) is a low-density parity-check code for error correction.
[0040] <14> MAC (Multiply-Accumulate) is a multiply-accumulate operation.
[0041] <15> ZSTD, the Z standard compression algorithm.
[0042] The present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement it. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.
[0043] like Figure 1This diagram illustrates the structure of an in-memory computing acceleration system based on UFS NAND, as described in an embodiment of the present invention.
[0044] The system includes:
[0045] UFS NAND storage array module 1 is used for partitioned storage of large model parameters, parameter sparsity indexes, and intermediate data generated during the calculation of large models; wherein, the UFS NAND storage array module 1 adopts 3D TLC UFS 4.0 NAND flash memory with a capacity of 256GB~4TB, which can fully meet the storage needs of large models of different sizes.
[0046] The logic chip integration module 2, integrated on the logic chip, is used to evaluate the task's computing power requirements and parameter sparsity when a request to initiate a large model task is received. Based on the evaluation results, it obtains the corresponding data from the UFS NAND storage array module 1, uses the large model deployed locally to perform attention-accelerated calculations to obtain the output calculation results, or migrates to an external chip for calculations and writes the calculation results back.
[0047] In one embodiment, such as Figure 2 The UFS NAND storage array module 1 is divided into a large model parameter area, a parameter sparse index area, and an intermediate result cache area; wherein,
[0048] The large model parameter area is used to divide the large model into sub-partitions based on the attention heads, and to independently store the weight parameters of each attention head. For example, for a 7B LLM (such as the Transformer model), its 32 attention heads can correspond to 32 independent sub-partitions, each with a capacity of 21.875GB, used to store the key weight matrices such as Q, K, and V for that attention head. To meet the needs of dynamic updates of weight parameters, a 16KB page-level management mechanism is used within the sub-partitions to support efficient random read and write operations.
[0049] The parameter sparse index region is used to maintain a sparse index table, recording the physical addresses of valid (i.e., non-zero) parameters in the large model parameter region during sparse access. This design aims to optimize I / O performance by reducing unnecessary data loading. For example, during the LLM inference phase, the system can load only the activated attention head parameters. Specifically, a 32-bit register can be used to identify the address of valid parameters within each sub-partition. Simultaneously, a sparsity threshold (e.g., ≥60%) is set at the hardware level. When the sparsity of a sub-partition exceeds this threshold, the sparse processing subunit is triggered to automatically skip the calculation of zero-value parameters, thereby improving overall computational efficiency.
[0050] The intermediate result cache is used to temporarily store intermediate data generated during the calculation of large models, such as Q, K, and V matrices, to avoid redundant calculations and improve processing speed. To extend the lifespan of UFS NAND, this cache adopts a circular buffer design, which effectively reduces frequent erase and write operations on the same physical block through sequential writing and overwriting.
[0051] In one embodiment, such as Figure 2 The logic chip integration module 2 includes:
[0052] The UFS protocol-compatible control unit is communicatively connected to the UFS NAND storage array module 1 and is used to manage address mapping, bad blocks, and data verification.
[0053] The large model task scheduling unit is used to evaluate the task's computing power requirements and parameter sparsity when it receives a request to initiate a large model task, and generate corresponding local computing instructions or execute migration computing operations based on the evaluation results.
[0054] The large model-adaptive AI computing unit is used to read the valid parameters located by sparse index when it receives local computing instructions, and use the large model deployed locally to accelerate the computing through attention to obtain the computing results.
[0055] A high-speed data buffer unit is used to cache data read from the UFS NAND storage array module 1 and intermediate results generated by the large model-adaptive AI computing unit.
[0056] In one embodiment, the UFS protocol compatible control unit supports the UFS 3.0 / 4.0 protocol standard, ensuring compatibility with mainstream UFS storage devices while fully leveraging the performance advantages of higher version protocols. Core functions include address mapping management of the NAND storage array, maintaining a partitioned mapping table, supporting fast address mapping queries and updates, and ensuring real-time data access. Bad block detection and isolation: Read and write tests are performed on all physical blocks of the NAND storage array at preset intervals. Potential bad blocks and damaged blocks are identified by judging read / write response time, data integrity, and other indicators. When other units initiate data access requests, the status identifier of the target physical block is first verified. If it is a marked bad block, address redirection is directly triggered, routing the data request to a spare, intact physical block. All detected bad blocks are written to a dedicated bad block list and permanently removed from the address mapping table, ensuring that subsequent data allocation and access completely avoid bad blocks and guaranteeing storage reliability. LDPC ECC verification: A hardware-level LDPC (Low-Density Parity-Check) error correction mechanism is adopted, specifically optimized for the bit error characteristics of NAND flash memory.
[0057] For address mapping management of NAND storage arrays, a sparse parameter index subunit is also provided, which locates the addresses of valid parameters in the sparse parameter index area through hardware. This subunit is a hardware module specifically designed for large model parameter sparsity scenarios. Through hardware-level fast addressing optimization, it solves the problems of low addressing efficiency and excessive invalid data transmission in traditional software.
[0058] In one embodiment, the UFS NAND storage array module 1 is connected to the logic chip via 64 high-speed vertical interconnect channels. Each channel has a speed of 2Gbps, resulting in a total bandwidth of 128Gbps, suitable for the high bandwidth requirements of large-scale models.
[0059] In one embodiment, such as Figure 3 The large-model-adaptive AI computing unit includes:
[0060] A highly parallel MAC array is designed to provide core multiply-accumulate operations for large models, specifically adapted for linear transformation scenarios such as fully connected layers. MAC (multiply-accumulate) is a core operation in linear transformation scenarios such as fully connected layers and convolutional layers of large models (e.g., multiplying and merging weight parameters with input features). This array is designed with ultra-high parallelism and optimized for massive multiply-accumulate operation requirements: preferably, it adopts a 1024x1024 two-dimensional array architecture, containing 1,048,576 independent MAC operation units (1024 rows × 1024 columns), supporting parallel multiply-accumulate operations with multiple inputs and multiple weights: it can complete 1,048,576 multiply-accumulate operations simultaneously in a single clock cycle, that is, output 1024 output features in one clock cycle. Each output feature is composed of the multiply-accumulate result of 1024 input features and corresponding weights, completely solving the computing power bottleneck of traditional serial operations. It is compatible with three mainstream precisions: INT8, FP16, and FP8, and can be dynamically switched according to task requirements: For low-precision scenarios (such as real-time inference and computing power-constrained scenarios): INT8 or FP8 precision is used, which further improves the computing speed and reduces power consumption while ensuring the availability of results. The computing power density of FP8 precision is twice that of FP16. For high-precision scenarios (such as model fine-tuning and precision-sensitive inference tasks): FP16 precision is used, at which time the array computing power can reach 800 GOPS (1 GOPS = 1 billion floating-point operations / second), which meets the massive computing needs of 7B~13B scale LLM fully connected layers.
[0061] The attention computation acceleration subunit is used to implement the core operations of large model attention mechanisms in hardware. Among these, the core supports hardware acceleration for three key operations: QKV matrix parallel multiplication: projecting the query (Q), key (K), and value (V) matrices and performing subsequent operations... , Matrix multiplication is performed in parallel via hardware circuitry, avoiding software-level loop iterations and improving computational efficiency. Scaled Dot-Product: The hardware directly performs the scaling operation by dividing by the square root of the key dimension, eliminating the need for additional software scheduling and reducing scaling latency. Softmax activation: Softmax computation is implemented using a hardware pipeline architecture, with parallel processing of exponents, summation, and normalization in three steps, avoiding numerical overflow issues in software implementations and reducing Softmax computation latency to the microsecond level. It also supports dynamic configuration of 1 to 64 attention heads, flexibly adapting to different needs of 7B to 13B scale LLMs. Compared to traditional software implementations, attention computation latency is reduced by 60%, significantly improving the response speed of model inference and training.
[0062] The sparse data processing subunit is used to locate valid parameter addresses based on sparse indexes and perform precise "zero-value skipping" processing on sparse data in the parameter region of a large model. Specifically, during the parameter reading phase, the sparse parameter indexing subunit hardware locates the addresses of valid parameters in the sparse parameter index region, automatically skipping invalid zero-value parameter addresses and reading valid parameter data, thus avoiding zero-value data consuming transmission bandwidth. During the computation phase, the MAC array and attention computation acceleration subunit only perform operations on valid parameters; zero-value parameters directly skip multiplication, addition, activation, and other operations, without consuming hardware computational resources. For sparse scenarios such as inactive attention head parameters in LLM, this subunit can automatically filter zero-value data, reading and processing only valid parameters, avoiding invalid computations and data transmission overhead, improving parameter access efficiency by 50%, and further optimizing overall computational performance.
[0063] In one embodiment, the large model task scheduling unit, as the core of overall decision-making for large model tasks, is responsible for accurately assessing task characteristics and dynamically selecting the optimal computing mode to achieve efficient collaboration between local computing power and external resources. Its specific functions are as follows: After receiving a large model task request from an external source, this unit will perform two core assessments: one is a task computing power requirement assessment, and the other is a parameter sparsity assessment. Based on the results of these two assessments, it will perform hierarchical task scheduling.
[0064] The task computing power requirement assessment clearly defines lightweight computing power (e.g., ≤800 GOPS) and intensive computing power (e.g., >800 GOPS). Lightweight computing power scenarios are typically exemplified by 7B LLM inference tasks, while intensive computing power scenarios are typically exemplified by 13B LLM fine-tuning tasks. The parameter sparsity assessment uses 60% as a threshold to divide high sparsity (e.g., ≥60%) and low sparsity (e.g., <60%), which works in conjunction with the hardware threshold for the parameter sparse index area.
[0065] If the current large model task satisfies "high sparsity and lightweight computing power" (such as high sparsity 7B LLM inference), then local computing instructions are directly generated, and large model-adaptive computing units are scheduled to perform local operations, giving full play to the advantages of local sparse processing and high parallel computing.
[0066] If the current large model task meets the requirements of "low sparsity and dense computing power" (such as low sparsity 13B LLM fine-tuning), then a migration computing operation is triggered to divert the task to an external high-computing-power node, avoiding performance bottlenecks caused by insufficient local computing power and ensuring efficient task progress.
[0067] In one embodiment, the migration computation operation includes: compressing relevant parameters using the ZSTD algorithm, migrating them to an external chip for processing via a high-speed UFS interface, so that the external chip can write back the computation results or necessary intermediate data to local storage. Specifically, the relevant task parameters are first compressed using the ZSTD compression algorithm (compression ratio up to 3:1), and then the compressed parameters are migrated to an external chip such as an edge NPU / AP for processing using a UFS 4.0 high-speed interface (transmission rate 16GB / s). After the external chip completes the computation, the final computation results or necessary intermediate data can be written back to the local UFS NAND storage array through the same high-speed interface. The entire migration process, through the combination of compression and high-speed transmission, significantly reduces data migration time and ensures task processing efficiency.
[0068] In a preferred embodiment, the high-speed data buffer unit uses 8MB of 8-port RAM as its core storage component, achieving a read / write speed of up to 8GB / s. Leveraging this characteristic, the unit can simultaneously cache 8 sets of attention head parameters and 4 sets of intermediate results (e.g., Q, K, V matrices). This efficient caching mechanism effectively avoids idle computing resources, ensuring the stability and improvement of the overall system's computational efficiency.
[0069] To better describe the UFS NAND-based in-memory computing acceleration system, specific embodiments are described below.
[0070] Example 1: 7B LLM inference task.
[0071] This scenario targets a large 7B-parameter model inference task with high sparsity (70%), requiring 600 GOPS of computing power (≤800 GOPS lightweight computing power threshold). A typical application is edge-side text generation (input text length 64 tokens, activating 8 attention heads). After the edge mobile processor initiates an inference request to the logic chip integrated module, the system starts a full-process collaborative computation: the large model task scheduling unit first parses the task information, and then performs a two-dimensional evaluation by combining the sparse data (70%) returned by the UFS protocol compatible control unit. If the condition of "high sparsity and lightweight computing power" is met, the local computation instruction is triggered.
[0072] After receiving the scheduling command, the protocol control unit quickly locates the valid parameter addresses corresponding to the eight activated attention heads through the hardware-level sparse parameter indexing subunit, filtering out 70% of zero-value invalid parameters, and finally extracting only 400MB of core valid parameters. These parameters are transmitted from the UFS NAND storage array to the high-speed data buffer unit through 64 high-speed vertical interconnect channels, and output to the AI computing unit in real time. The attention computing acceleration subunit performs QKV matrix multiplication (1.2ms) and Softmax activation operation (0.3ms) in parallel, and a 1024x1024 high-parallel MAC array synchronously completes the fully connected layer multiply-add operation (1.5ms). The three components work together in a pipeline without any data waiting gaps. The total computation time is 3.8ms. The final calculation result is sent back to the mobile phone processor via the UFS 4.0 protocol interface to complete the text generation and output.
[0073] Example 2: 13B LLM fine-tuning task.
[0074] This scenario focuses on a fine-tuning task of a large 13B model with low sparsity (30%), requiring 1200 GOPS of computing power (far exceeding the 800 GOPS threshold for lightweight computing power). A typical application is small-sample domain adaptation on edge devices (batch size 16, requiring updates to 16 attention head parameters). After the edge device processor initiates a fine-tuning request to the logic chip integrated module, the large model task scheduling unit immediately initiates a two-dimensional evaluation: combining the task batch size and parameter update range, the total computing power is calculated to reach 1200 GOPS. At the same time, the parameter sparsity data (30%) is obtained through the UFS protocol compatible control unit, determining that the "low sparsity and dense computing power" condition is met, triggering the migration computation operation.
[0075] After receiving the migration command, the UFS protocol-compatible control unit accurately reads the full parameters corresponding to the 16 target attention heads (only 30% of invalid data is filtered due to low sparsity, resulting in a total of 6.4GB of valid parameters). The built-in ZSTD compression engine compresses the parameters to 2.1GB, significantly reducing migration transmission overhead. The compressed parameters are then rapidly transmitted to the high-speed data buffer unit via the UFS 4.0 protocol interface (total bandwidth 16GB / s), and forwarded to the edge device NPU (4TOPS computing power). Upon receiving the data, the NPU decompresses and restores the parameters, initiating the fine-tuning training process. Intermediate results generated during training are temporarily stored in the NPU's local DRAM to avoid frequent interaction with the original storage array. After fine-tuning, the NPU compresses the updated 6.4GB parameters again and sends them back to the high-speed data buffer unit. After decompression and verification by the UFS protocol-compatible control unit, the parameters are accurately written back to the parameter area of the UFS NAND storage array, while simultaneously updating the parameter sparsity index. The entire migration computation process relies on high-bandwidth interfaces and data compression optimization to effectively avoid the bottleneck of insufficient local computing power. The total time is only 420ms, which not only ensures the efficient progress of fine-tuning tasks, but also reduces the overall power consumption through compressed transmission and cache scheduling, perfectly adapting to the intensive model optimization needs of edge devices.
[0076] This invention also provides an electronic device that utilizes the UFS NAND-based in-memory computing acceleration system described in the above embodiments. With this system, the electronic device can achieve low-latency response during local inference of large models and efficient migration and operation when performing intensive fine-tuning tasks. Simultaneously, through a series of optimization techniques such as sparse data filtering, high-bandwidth transmission, and mixed-precision computing, the electronic device can significantly reduce the power consumption of AI computations. The integration of this in-memory computing acceleration system allows the electronic device to efficiently meet the core computational needs of large models ranging from 7B to 13B without relying on cloud computing power. This not only significantly improves the execution efficiency, response speed, and energy efficiency of edge-side AI tasks but also endows the electronic device with stronger AI computation autonomy, enabling it to better adapt to various application scenarios. Such electronic devices cover a wide range, including but not limited to mobile phones, computer equipment, tablet devices, personal digital processing devices, and factory back-end processing equipment.
[0077] Compared with the prior art, the present invention has the following advantages:
[0078] 1. Precisely matching the characteristics of large models, with dual optimization of migration and energy consumption costs: This invention deeply adapts to the sparsity characteristics of large model parameters and the computational requirements of attention mechanisms. Through hardware-level sparse parameter access and attention acceleration design, the local computation ratio of 7BLLM inference reaches up to 90%, and the frequency of parameter migration across devices is reduced by 75%. Compared with traditional solutions, the single inference time is compressed from 15ms to 4ms, the response speed is improved by 3.75 times, the computational power consumption is reduced from 150mW to 50mW, and the energy efficiency ratio is optimized by 67%, perfectly balancing the requirements of low latency and low power consumption on the edge.
[0079] 2. Comprehensive computing power coverage and compatibility with multiple mainstream large models: This invention has a local FP16 precision computing power of 800 GOPS, and the computing power density is adapted to the core requirements of the edge side. It can fully cover inference tasks of 7B / 13B scale LLM, lightweight Transformer architecture image generation (such as Stable Diffusion Tiny), and other scenarios. It is compatible with mainstream AI model types on the edge side, without the need for separate adaptation for different models, thus improving the versatility of the system.
[0080] 3. Natively compatible with UFS protocol, with extremely low deployment and industrialization costs: This invention is based on the UFS 3.0 / 4.0 universal protocol architecture design, which can directly replace the UFS NAND storage module in existing electronic devices without modifying the device motherboard layout, interface design and drivers, achieving "plug and play" upgrade; during the industrialization process, the manufacturing cost increases by no more than 20% compared to traditional solutions, and it has extremely strong feasibility for mass production.
[0081] 4. Leading sparse processing efficiency and significantly reduced bus and storage overhead: The proprietary sparse data processing subunit of this invention improves the access efficiency of large model parameters by 50% through hardware-level "zero value skipping" and fast addressing mechanisms, effectively avoiding redundant data transmission of invalid data; at the same time, it reduces the system bus bandwidth usage by 40%, alleviates the pressure on the end-side bus transmission, further optimizes the overall computing efficiency, and fully adapts to the core characteristics of large model parameter sparsity.
[0082] In summary, the UFS NAND-based in-memory computing acceleration system and electronic device of this invention utilizes a UFS NAND storage array module to partition and store large model parameters, parameter sparsity indexes, and intermediate data generated during large model computation. Upon receiving a request to initiate a large model task, the logic chip integration module assesses the task's computing power requirements and parameter sparsity. Based on the assessment results, it retrieves the corresponding data from the UFS NAND storage array module and uses the locally deployed large model for attention-accelerated computation to obtain the output computation result, or migrates the computation to an external chip and writes the result back. This invention is highly adaptable to the core characteristics of large models, supports parameter sparse access and hardware-accelerated attention mechanisms, and can meet the computational needs of large models such as LLM and Transformer. Furthermore, it enhances the computing power of UFS NAND logic chips, is compatible with the UFS protocol and existing architectures, and can directly replace edge devices without hardware modifications, reducing costs. It also optimizes task scheduling, intelligently switching between local computation and external migration based on task type and parameter sparsity, balancing efficiency and performance. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.
[0083] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.
Claims
1. A memory computing acceleration system based on UFS NAND, characterized in that, include: UFS NAND storage array modules are used for partitioned storage of large model parameters, parameter sparsity indexes, and intermediate data generated during large model calculations. The logic chip integration module, connected to the UFS NAND storage array module, is used to evaluate the task's computing power requirements and parameter sparsity when a request to initiate a large model task is received. Based on the evaluation results, it obtains the corresponding data from the UFS NAND storage array module, uses the large model deployed locally for attention acceleration calculation to obtain the output calculation results, or migrates to an external chip for calculation and writes the calculation results back. The logic chip integration module includes: UFS protocol-compatible control unit, used to manage address mapping, bad blocks, and data verification; The large model task scheduling unit is used to evaluate the task's computing power requirements and parameter sparsity when it receives a request to initiate a large model task, and generate corresponding local computing instructions or execute migration computing operations based on the evaluation results. The large model-adaptive AI computing unit is used to read the valid parameters located by sparse index when it receives local computing instructions, and use the large model deployed locally to accelerate the computing through attention to obtain the computing results. A high-speed data buffer unit is used to cache data read from the UFS NAND storage array module and intermediate results generated by the large model-adaptive AI computing unit.
2. The in-memory computing acceleration system based on UFS NAND as described in claim 1, characterized in that, The UFSNAND storage array module is divided into a large model parameter area, a parameter sparse index area, and an intermediate result cache area; among which... The large model parameter area is used to divide the large model into sub-partitions according to each attention head, so as to store the weight parameters of the corresponding attention head respectively; The parameter sparse index area is used to record the addresses of valid parameters when parameters are sparsely accessed; The intermediate result cache is used to temporarily store intermediate data generated during the calculation of large models.
3. The in-memory computing acceleration system based on UFS NAND as described in claim 1, characterized in that, The UFS protocol compatible control unit supports UFS 3.0 and / or 4.0 protocols, and also has a sparse parameter index subunit, which locates the address of the valid parameter in the sparse parameter index area through hardware.
4. The in-memory computing acceleration system based on UFS NAND as described in claim 1, characterized in that, The large model-adaptive AI computing unit includes: Highly parallel MAC arrays are used to provide multiply-accumulate capabilities for large models to accommodate linear transformations of large models; The attention computation acceleration subunit is used to implement the core operations of the attention mechanism for large models using hardware; wherein, the core operations include: QKV matrix multiplication, Scaled Dot-Product, and Softmax operation; The sparse data processing subunit is used to perform zero-value skipping processing on the sparse data in the large model parameter area based on the address of the valid parameters located by the sparse index, and only read the valid parameters.
5. The in-memory computing acceleration system based on UFS NAND as described in claim 1, characterized in that, The large model task scheduling unit is used to evaluate the computing power requirements and parameter sparsity of the current large model task. For tasks with high sparsity and low computational power, generate local computation instructions; For tasks with low sparsity and high computational power, a migration computation operation is performed.
6. The in-memory computing acceleration system based on UFS NAND as described in claim 5, characterized in that, The migration calculation operation includes: compressing the relevant parameters using the ZSTD algorithm, migrating them to an external chip for processing via a high-speed UFS interface, so that the external chip can write back the calculation results or the required intermediate data to local storage.
7. The in-memory computing acceleration system based on UFS NAND as described in claim 2, characterized in that, The UFSNAND storage array module is connected to the logic chip via 64 high-speed vertical interconnect channels.
8. The in-memory computing acceleration system based on UFS NAND as described in claim 1, characterized in that, The high-speed data buffer unit uses 8MB of 8-port RAM.
9. An electronic device, characterized in that, The application uses the UFS NAND-based in-memory computing acceleration system as described in any one of claims 1 to 8.