Systolic array architecture, in-memory compute architecture, and vision model acceleration system
By combining a pulsating array architecture with non-volatile memory, logical operations can be performed within the memory cell, solving the von Neumann bottleneck problem, reducing chip area and power consumption, supporting higher operating frequencies, and achieving higher area efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTHERN UNIVERSITY OF SCIENCE AND TECHNOLOGY
- Filing Date
- 2026-01-30
- Publication Date
- 2026-06-05
AI Technical Summary
In existing storage computing architectures, the physical separation of storage units and computing units leads to data migration that consumes far more energy and time than the computing itself, resulting in the von Neumann bottleneck, which limits system energy efficiency. Furthermore, existing storage media cannot deploy large-scale model weights at high density within limited on-chip space, resulting in large chip area and power consumption, and low operating frequency.
A pulsating array architecture is adopted to broadcast input data within the computing unit group for parallel processing, and transmit data between groups in a pulsating manner. Logical operations are performed by combining an XOR array and an adder tree module, which reduces signal fan-out load and pipeline register usage. Non-volatile memory such as resistive random access memory is used for weight resident mode.
It reduces chip area and power consumption, supports higher operating frequencies, achieves higher area efficiency, and balances computing density and frequency through intra-group parallelism and inter-group pulsation design, breaking through the limitations of traditional architecture.
Smart Images
Figure CN122152372A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of microelectronics, and more particularly to a pulsating array architecture, an in-memory computing architecture, and a visual model acceleration system. Background Technology
[0002] With the development of artificial intelligence technology, especially the widespread adoption of large language models and high-precision vision tasks, the data throughput faced by computing systems is growing exponentially. Under the mainstream von Neumann architecture, the physical separation of storage and computing units forces massive amounts of data to repeatedly migrate between the two. The energy and time consumed by data transmission far exceed the computation itself, and this "von Neumann bottleneck" has become the primary factor restricting system energy efficiency.
[0003] To overcome the aforementioned architectural bottlenecks, Computing-in-Memory (CIM) technology offers a paradigm shift solution. Unlike traditional architectures, CIM eliminates the physical boundary between computing and storage units by performing logical or arithmetic operations in-situ within the storage array. By simultaneously activating multiple word lines in the storage array, it efficiently performs large-scale matrix-vector multiplications using the physical laws of the analog domain (such as current summation) or near-memory logic circuits in the digital domain. This massively parallel processing approach not only avoids frequent movement of weighted data, significantly reducing the high power consumption and latency caused by data transmission, but also greatly improves the system's computing density, providing an ideal pathway to achieve energy-efficient edge AI computing.
[0004] Existing CIM architectures have explored different technical approaches based on various storage media. Volatile memories, represented by SRAM and DRAM, typically maintain the array topology, multiplex read / write circuits, and integrate logic units inside or near the array. They efficiently reconstruct vector-matrix multiplication through bitwise logic operations or local bit-level addition, combined with peripheral digital accumulation logic. On the other hand, non-volatile media such as embedded Flash and phase-change memory utilize the multi-level threshold voltage or conductance characteristics of the devices to simultaneously activate multiple rows of cells and directly complete analog or quasi-analog MAC operations using the superposition characteristics of bit line currents. MRAM also utilizes magnetoresistive state differences to modulate the read current to achieve parallel computing. The aforementioned solutions all have their limitations: SRAM and DRAM are limited by their large cell area and volatility, making it difficult to deploy large-scale model weights at high density within a limited on-chip space; Flash faces high power consumption and long programming / erasing latency, and has limited write / erase endurance; PCM has high write power consumption and a relatively longer write latency than volatile memory, and although its endurance is better than Flash, it is still a finite-life device, making it difficult to support frequent online learning or fine-tuning tasks at the edge; while MRAM's small resistor-to-switching ratio limits the read margin and computational linearity of large-scale arrays. Therefore, when existing mainstream memory media are integrated into chips, the resulting chip area and power consumption are large, and the operating frequency is low. Summary of the Invention
[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a pulsating array architecture, in-memory computing architecture, and visual model acceleration system to reduce chip area and power consumption, support higher operating frequencies, and achieve higher area efficiency.
[0006] The technical solution of the present invention is as follows: A pulsating array architecture, comprising: Multiple processing unit groups, each of the processing unit groups including multiple parallel processing units; Each of the processing unit groups integrates a memory unit for storing weights, and the processing unit groups adopt a weight-resident mode. Multiple parallel processing units are used to perform calculations on the input data according to weights; Within each clock cycle, each processing unit group broadcasts the input activation vector directly to multiple parallel processing units within each processing unit group according to a unified packet format. Among the multiple processing unit groups, the input activation data is passed step by step in a pulsating manner.
[0007] Optionally, the parallel processing unit includes: An XOR array, configured with XOR logic circuitry, is used to perform XOR logic operations on input data according to weights to generate multi-way partial products. The addition tree module is used to perform spatial dimension reduction and summation on the multi-way partial products generated by logic circuit operations.
[0008] Optionally, the memory unit is one of resistive random access memory, magnetoresistive random access memory, phase change memory, or ferroelectric field-effect transistor.
[0009] The present invention also proposes an in-memory computing architecture, including an input module, an output module, and a systolic array architecture as described above. The input module is used to output raw data to the systolic array architecture, and the output module is used to output the data output by the systolic array architecture to an external module.
[0010] Optionally, the input module includes: An asynchronous first-in-first-out queue is used to output raw data from external modules after bandwidth matching. An input buffer is used to cache the data output by the asynchronous first-in-first-out queue; The activation value conversion module is used to parse and convert the floating-point format data output from the input buffer into a tail value. The activation value pulsation control module performs vectorized recombination and distribution of the bit values output by the activation value conversion module based on the group vector pulsation strategy.
[0011] Optionally, the output module includes: A shift adder is used to shift and add the data output by the pulsating array architecture before outputting it. An accumulator is used to accumulate the data output from the shift adder and then output the result. The output buffer is used to cache the data output by the accumulator for external modules to read.
[0012] This invention also proposes a data processing method applied to the above-mentioned in-memory computing architecture, comprising the following steps: Define a four-dimensional data format: [CHin+Tin-1) / Tin, H, W, Tin], where the data is arranged in this format, where CH is the original number of channels, H and W are the spatial dimensions of the feature map, and Tin is the input parallelism of the hardware. A dimension folding strategy is adopted for the multi-head self-attention operator, which folds and maps any logical dimension other than the channel dimension to the spatial dimension H.
[0013] Optionally, the data processing method further includes the following steps: Before loading, the input data is divided into blocks according to the input parallelism and organized into the shape of [CHinTin, H, W, Tin]. Each clock cycle, a data packet containing Tin elements is read.
[0014] The present invention also proposes a visual model acceleration system, including the in-memory computing architecture described above.
[0015] Optionally, the visual model acceleration system also includes: The controller stores a data processing program, which, when executed, performs the data processing method described above on the in-memory computing architecture.
[0016] This invention employs a systolic array architecture comprised of multiple processing unit groups. Each processing unit group includes multiple parallel processing units. Each processing unit group integrates a memory unit for storing weights, and the processing unit groups adopt a weight-resident mode. The multiple parallel processing units perform calculations on the input data according to the weights. Within each clock cycle, each processing unit group broadcasts the input activation vector directly to the multiple parallel processing units within it, using a unified packet format. The input activation data is transmitted sequentially between the multiple processing unit groups in a systolic manner. This systolic array architecture design cuts off global longlines, reduces signal fan-out load, decreases the number of pipeline registers used, reduces chip area and power consumption, supports higher operating frequencies, and achieves higher area efficiency. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0018] Figure 1 This is a functional module diagram of an embodiment of the pulsating array architecture of the present invention.
[0019] Figure 2 This is a schematic diagram of the functional modules of an embodiment of the in-memory computing architecture of the present invention.
[0020] Figure 3 This is a schematic diagram of the hardware mapping of the in-memory computing architecture of the present invention to the neural network.
[0021] Figure 4 This is a schematic diagram of the microarchitecture of the parallel processing unit in the pulsating array architecture of this invention.
[0022] Figure 5This is a timing diagram of an embodiment of the pulsating array architecture of the present invention.
[0023] Figure 6 This is a flowchart of the method steps of an embodiment of the data processing method of the present invention.
[0024] Explanation of reference numerals in the attached figures: 100, processing unit group; 10, parallel processing unit; 11, XOR array; 12, addition tree module; 20, asynchronous first-in-first-out queue; 30, input buffer; 40, activation value conversion module; 50, activation value pulsation control module; 60, shift adder; 70, accumulator; 80, output buffer. Detailed Implementation
[0025] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0026] In the implementation methods and claims, unless otherwise specified in the text, the terms "a," "an," "the," and "the" may also include plural forms. If the embodiments of the present invention involve descriptions of "first," "second," etc., such descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features.
[0027] It should be further understood that the term "comprising" as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when an element is referred to as "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements present. Furthermore, "connected" or "coupled" as used herein can include wireless connections or wireless coupling. The term "and / or" as used herein includes all or any unit and all combinations of one or more associated listed items.
[0028] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0029] Furthermore, the technical solutions of the various embodiments can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.
[0030] With the development of artificial intelligence technology, especially the widespread adoption of large language models and high-precision vision tasks, the data throughput faced by computing systems is growing exponentially. Under the mainstream von Neumann architecture, the physical separation of storage and computing units forces massive amounts of data to repeatedly migrate between the two. The energy and time consumed by data transmission far exceed the computation itself, and this "von Neumann bottleneck" has become the primary factor restricting system energy efficiency.
[0031] To overcome the aforementioned architectural bottlenecks, Computing-in-Memory (CIM) technology offers a paradigm shift solution. Unlike traditional architectures, CIM eliminates the physical boundary between computing and storage units by performing logical or arithmetic operations in-situ within the storage array. By simultaneously activating multiple word lines in the storage array, it efficiently performs large-scale matrix-vector multiplications using the physical laws of the analog domain (such as current summation) or near-memory logic circuits in the digital domain. This massively parallel processing approach not only avoids frequent movement of weighted data, significantly reducing the high power consumption and latency caused by data transmission, but also greatly improves the system's computing density, providing an ideal pathway to achieve energy-efficient edge AI computing.
[0032] Existing CIM architectures have explored different technical approaches based on various storage media. Volatile memories, represented by SRAM and DRAM, typically maintain the array topology, multiplex read / write circuits, and integrate logic units inside or near the array. They efficiently reconstruct vector-matrix multiplication through bitwise logic operations or local bit-level addition, combined with peripheral digital accumulation logic. On the other hand, non-volatile media such as embedded Flash and phase-change memory utilize the multi-level threshold voltage or conductance characteristics of the devices to simultaneously activate multiple rows of cells and directly complete analog or quasi-analog MAC operations using the superposition characteristics of bit line currents. MRAM also utilizes magnetoresistive state differences to modulate the read current to achieve parallel computing. The aforementioned solutions all have their limitations: SRAM and DRAM are limited by their large cell area and volatility, making it difficult to deploy large-scale model weights at high density within a limited on-chip space; Flash faces high power consumption and long programming / erasing latency, and has limited write / erase endurance; PCM has high write power consumption and a relatively longer write latency than volatile memory, and although its endurance is better than Flash, it is still a finite-life device, making it difficult to support frequent online learning or fine-tuning tasks at the edge; while MRAM's small resistor-to-switching ratio limits the read margin and computational linearity of large-scale arrays. Therefore, when existing mainstream memory media are integrated into chips, the resulting chip area and power consumption are large, and the operating frequency is low.
[0033] To address the aforementioned problems, this invention proposes a pulsating array architecture.
[0034] Reference Figure 1 and Figure 2 In one embodiment, the pulsating array architecture includes: Multiple processing unit groups 100, each of the processing unit groups 100 including multiple parallel processing units 10; Each of the processing unit groups 100 integrates a memory unit for storing weights, and the processing unit groups 100 adopt a weight-resident mode. The multiple parallel processing units 10 are used to perform calculations on the input data according to the weights; In each clock cycle, each processing unit group 100 broadcasts the input activation vector directly to the multiple parallel processing units 10 within each processing unit group 100 according to a unified packet format. Among the multiple processing unit groups 100, the input activation data is passed step by step in a pulsating manner.
[0035] In this embodiment, a systolic array architecture is used to address the efficiency issues of traditional computing architectures in Resistive Random Access Memory (RRAM) array applications. It should be noted that while traditional systolic arrays offer high frequencies, they require a large number of pipelined registers, resulting in significant area and power consumption overhead. Conversely, vector arrays, while requiring fewer registers, suffer from excessive fan-out load under high computing power, limiting their operating frequency. To balance these two aspects, this invention divides the computing array into multiple processing unit groups 100 (PE Groups). In practice, the RRAM array utilizes its non-volatile storage weights, broadcasting input data within the same processing unit group 100 to complete multi-path parallel computation within a single cycle. Between different processing unit groups 100, data is transmitted in a systolic manner. This "intra-group parallelism, inter-group systolic" design leverages the high-density computing advantages of RRAM while significantly reducing register usage through hierarchical data flow, and simultaneously reducing the signal fan-out pressure caused by global broadcasting.
[0036] Specifically, this solution logically and physically divides the large-scale computing array into multiple processing unit groups 100. For example, 64 parallel processing units 10 can be divided into two groups of 32 units each. The specific number of processing unit groups 100 and the number of parallel processing units 10 in each processing unit group 100 can be set according to actual conditions and user needs. Each processing unit group 100 can integrate RRAM cells for storing weights and adopt a weight-resident mode. This solution establishes a hybrid data flow mechanism: within a single processing unit group 100, the input vector is directly broadcast to all parallel processing units 10 within the group, using short-distance broadcasting to achieve high parallelism similar to a vector machine; while between different processing unit groups 100, input data or partial data is passed step-by-step through registers in a pulsating manner. This design of this solution cuts off the global long pipeline, significantly reduces the signal fan-out load, and reduces the number of pipeline registers used compared to traditional fully pulsating arrays. The final effect is to reduce chip area and power consumption, support higher operating frequencies, and achieve higher area efficiency.
[0037] The technical solution of this invention uses a systolic array architecture composed of multiple processing unit groups 100, wherein each processing unit group 100 includes multiple parallel processing units 10; each processing unit group 100 integrates a memory unit for storing weights, and the processing unit group 100 adopts a weight-resident mode or a weight systolic mode; the multiple parallel processing units 10 are used to calculate and process the input data according to the weights; in each clock cycle, each processing unit group 100 directly broadcasts the input activation vector to the multiple parallel processing units 10 within each processing unit group 100 according to a unified packet format; among the multiple processing unit groups 100, the input activation data is passed step by step in a systolic manner. Thus, the design of the systolic array architecture of this solution cuts off the global long line, reduces the signal fan-out load, reduces the number of pipeline registers used, reduces chip area and power consumption, supports higher operating frequencies, and achieves higher area efficiency.
[0038] Reference Figure 2 In one embodiment, the parallel processing unit 10 includes: XOR array 11 is configured with XOR logic circuitry. XOR array 11 is used to perform XOR logic operation on input data according to weights to generate multi-way partial product. The addition tree module 12 is used to perform spatial dimension reduction summation on the multi-way partial products generated by logic circuit operations.
[0039] In this embodiment, multiple parallel processing units 10 are integrated within each processing unit group 100. These parallel processing units 10 are physically arranged closely and configured in an intra-group broadcast mode, meaning that all parallel processing units 10 within the group share the same set of activation value inputs, but each has its own independent weight storage space. Each parallel processing unit 10 contains an XOR array 11, which utilizes the high-resistivity and low-resistivity physical characteristics of the RRAM device to store 1-bit quantization weights and is configured with digital XOR logic circuits to replace multipliers, performing parallel bit operations directly at the storage array. An adder tree module 12 is connected below the RRAM array, which can be used to perform fast spatial dimension reduction and summation of the multi-way partial products generated by the XOR logic circuit operations.
[0040] In one embodiment, the memory cell is one of resistive random access memory, magnetoresistive random access memory, phase-change memory, or ferroelectric field-effect transistor.
[0041] In this embodiment, in addition to RRAM, the memory cells can also use SRAM, DRAM, eDRAM, or other new non-volatile memories (such as MRAM, PCM, FeFET, Flash). Although the physical read / write mechanisms of individual memory cells differ in these media, the array composed of them can still be divided into processing unit groups 100 and perform intra-group broadcasting and inter-group pulsed data scheduling.
[0042] This invention also proposes an in-memory computing architecture.
[0043] Reference Figure 2 In one embodiment, the in-memory computing architecture includes an input module, an output module, and a systolic array architecture as described above. The input module is used to output raw data to the systolic array architecture, and the output module is used to output the data output by the systolic array architecture to an external module.
[0044] Reference Figure 2 In one embodiment, the input module includes: Asynchronous first-in-first-out queue 20 is used to output the raw data from external modules after bandwidth matching; Input buffer 30 is used to buffer the data output by the asynchronous first-in-first-out queue 20; The activation value conversion module 40 is used to parse and convert the floating-point format data output by the input buffer 30 into a mantissa value. The activation value pulsation control module 50 performs vectorized recombination and distribution of the bit values output by the activation value conversion module 40 based on the group vector pulsation strategy.
[0045] Reference Figure 2 In one embodiment, the output module includes: The shift adder 60 is used to shift and add the data output by the pulsating array architecture before outputting it. Accumulator 70 is used to accumulate the data output by shift adder 60 and then output it. The output buffer 80 is used to buffer the data output by the accumulator 70 for external modules to read.
[0046] The in-memory computing architecture proposed in this embodiment is based on resistive random access memory (RRAM). This architecture employs a hierarchical array design, comprising three main parts from top to bottom: an input module, a systolic array architecture, and an output module. Through the coordinated operation of digital peripheral circuits, it achieves a mixed-precision heterogeneous computing paradigm of BF16 (activation value) × INT1 (weight). The overall system of the in-memory computing architecture aims to efficiently implement in-memory matrix multiplication operations with BF16 inputs and 1-bit weights.
[0047] In the structural design of the core in-memory computing array, this invention adopts a three-level hierarchical structure of processing unit group 100—processing unit—array. Regarding the data flow and logical control of key modules, this scheme constructs a data processing link of 'input preprocessing—pulse distribution—in-memory computation—result reduction'. Input stage: Raw data from an external high-speed interface (such as an AXI bus) first enters the asynchronous first-in-first-out queue 20 to solve the bandwidth matching problem across clock domains, and is then temporarily stored in the input buffer 30 to support data multiplexing and reading. Preprocessing stage: Data enters the activation value conversion module 40. Since the input is in BF16 floating-point format, this module is responsible for parsing the floating-point data, stripping the exponent, and converting the BF16 format activation value into a 12-bit mantissa value, preparing for subsequent fixed-point computation. Distribution stage: The converted data is taken over by the activation value pulse control module 50. This module is designed based on a group vector pulse strategy, and its core function is to vectorize and reorganize the activation value data and distribute it. Specifically, this module arranges the preprocessed activation value data into a feature vector that matches the number of parallel processing units 10 (e.g., 32) within the processing unit group 100. During data transmission, this feature vector is simultaneously and in parallel sent to all parallel processing units 10 within the processing unit group 100, ensuring that each computing node within the group can obtain the complete activation value vector data at the same time, thereby maximizing the utilization of the array's parallel computing capabilities and matching the throughput requirements of the underlying RRAM array.
[0048] like Figure 3 As shown, this invention also proposes an RRAM CIM array architecture and mapping scheme for Vision Transformer (ViT), which is specifically optimized at the hardware level for the inference process of the ViT model. The core of this embodiment lies in employing a hierarchical mapping strategy to precisely match the feature map dimensions at different stages of the ViT computation flow with a CIM array of specific physical dimensions, thereby maximizing hardware utilization and reducing energy consumption. Figure 3As shown in the flowchart on the left, the computation of the ViT model is divided into four stages: Stage 0 to Stage 3. Stage 0 is mainly responsible for Patch Embedding and preliminary convolution processing, with output feature map sizes of [28, 28, 128]. Subsequent Stages 1, 2, and 3 contain the main Transformer encoder module (ViT Block), with feature embedding dimensions of 128, 160, and 320 for each stage, respectively. To accommodate this dimensional variation, this invention designs three different RRAM CIM arrays: 128×128, 160×160, and 320×320. This customized array design directly matches the matrix operation requirements of each stage, effectively avoiding zero-padding waste when mapping matrices of different dimensions, thus significantly reducing power consumption and saving chip area at the hardware level.
[0049] In Stage 1, the input feature map has a dimension of 128. For example... Figure 3 As shown in the cyan area at the top right, this stage contains two repeating ViT Blocks. To map the weight matrix of this stage, the system uses a 128x128 CIM array. The specific mapping relationship is: Q, K, V weight matrices in the attention mechanism (W... QKV The logical size of the first layer (FC1) is 384×128, which is physically divided into three parallel 128×128 CIM arrays for storage and computation. The projection layer weight matrix is 128×128 in size and is directly mapped to one 128×128 CIM array. The first layer (FC1) weight of the multilayer perceptron (MLP) has a weight size of 512×128, which is split and mapped to four 128×128 CIM arrays. The second layer (FC2) weight has a weight size of 128×512, which is also mapped to four 128×128 CIM arrays through transposition or parallel read mechanisms. Through this physical parallel mapping, the system can complete the multiplication of multiple submatrices simultaneously within one clock cycle, significantly improving inference throughput.
[0050] Entering Stage 2, after downsampling, the feature map size becomes [14, 14, 160]. For example... Figure 3 As shown in the red area in the middle of the right side, this stage contains 6 repeating ViT Blocks. For the 160 embedding dimension, this embodiment uses a 160×160 CIM array. Wherein, W... QKVThe logical size (480×160) is mapped to three 160×160 CIM arrays; the Projection layer (logical size 160×160) is mapped to one array of the same size; the FC1 layer (logical size 640×160) is mapped to four arrays; and the FC2 layer is mapped to four arrays. This stage avoids the splicing complexity or area waste associated with using standard 128×128 or 256×256 arrays by using dedicated 160×160 arrays.
[0051] In Stage 3, the feature map is further downsampled to [7, 7, 320]. For example... Figure 3 As shown in the yellow area at the bottom right, this stage contains two ViT Blocks. For the 320 embedding dimension, the system employs a large 320×320 CIM array. Among them, W... QKV The logic size (960×320), Projection layer (320×320), FC1 layer (1280×320), and FC2 layer (320×1280) are mapped to 3, 1, 4, and 4 320×320 CIM arrays, respectively. This large-size array design not only ensures high parallelism but also simplifies the control logic.
[0052] Throughout all the aforementioned stages, the RRAM CIM array is configured to store only quantized (e.g., BF16×1-bit) static network weights, which are pre-written before inference begins and remain fixed during inference. Since write operations on RRAM devices typically have higher latency and power consumption than read operations, and are subject to tolerance limitations, this invention circumvents the write bottleneck of RRAM through a "write-once, read-many" in-memory computing model. Simultaneously, by leveraging its non-volatility and high-density characteristics, it eliminates the frequent transfer of weight data from off-chip DRAM in traditional architectures, thereby overcoming the memory wall limitation.
[0053] like Figure 4As shown, this invention also provides a parallel processing unit 10 based on resistive random access memory (RRAM). This unit is the core component connecting the upper-layer BF16 quantization algorithm and the lower-layer hardware array, and is mainly responsible for dynamic alignment of input data and high-precision in-memory calculation. In the input data preprocessing stage, the parallel processing unit 10 first receives a BF16 format input vector (128 input channels in this embodiment) from the upper-level buffer through the input interface. To facilitate subsequent processing, each input data is split into a sign bit (S), an exponent bit (E), and a mantissa bit (M). To address the alignment difficulties in traditional floating-point arithmetic, the parallel processing unit 10 integrates a comparison tree module. This module can compare the exponent parts (1 to 128) of all 128 input data in the current batch in parallel, thereby quickly determining the maximum exponent value (Emax) in the batch. Subsequently, the calculator and shifter module performs a right shift operation on the original mantissa according to the difference between the exponent of each input and the maximum exponent (ΔE=Emax-Ei), and performs necessary zero-padding and truncation to convert it into a 12-bit extended mantissa under a unified benchmark. This processing method, known as "batch granular alignment," differs from traditional layer granular alignment and can effectively preserve valid bits of data.
[0054] Figure 4 This paper demonstrates a microarchitecture of a parallel processing unit 10 specifically designed to support BF16 inputs and 1-bit static weight multiply-accumulate operations. First, in the input preprocessing stage, the parallel processing unit 10 receives and processes a batch of 128 BF16 formatted activation inputs. It extracts the maximum exponent value of the current batch using an internal comparison tree and calculator, calculates the exponent difference, and then performs high-precision batch-level mantissa alignment using a mantissa shifter with 4-bit guard bits. Subsequently, the aligned mantissa data is broadcast to the core computation region, which consists of 32 CIM arrays. Each array further contains 128 RRAM_CIM macrocells, which pre-store quantized 1-bit static weights using RRAM devices, performing large-scale parallel in-memory multiply-accumulate operations. Finally, the intermediate results generated by each array are aggregated through a partial summation and addition tree, and the corresponding bit weights are restored through a left shift operation. The results are then fed into an accumulator 70 for numerical summation and output in a standardized data frame format, allowing them to be directly used for the next layer's computation without additional rearrangement.
[0055] First, the batch alignment strategy greatly improves computational accuracy. Compared to the problem of traditional layer-level alignment where the mantissa is truncated due to the large dynamic range of the entire layer's data, the comparison tree and mantissa shifter used in this invention reduce the alignment range to the current computation batch, preserving the effective information of the mantissa to the maximum extent.
[0056] Secondly, the fully digital CIM architecture achieves excellent energy efficiency and robustness. The parallel processing unit 10 employs a digital addition tree and shift-accumulation logic, avoiding noise interference and non-idealities in signal accumulation found in analog CIM schemes, and eliminating the need for expensive analog-to-digital conversion. To verify the computational reliability of this architecture under real network loads, this embodiment selected a 128-rows × 128-column weight matrix from the Tiny-ViT network quantization weights and compared it with an input matrix of the same size for benchmark testing. The experiment compared the hardware computation results with the software simulation results. The average absolute error of the obtained 128 × 128 result matrix was only 0.009, and the average relative error was as low as 0.004. This extremely low error level indicates that the hardware computation results closely approximate the software simulation, fully demonstrating that this architecture is fully capable of meeting the high-precision computational requirements of real neural networks.
[0057] Furthermore, the fine-grained parallel architecture significantly improves throughput. By integrating 32 parallel CIM arrays in a single parallel processing unit 10 and combining them with a high-efficiency 4T-XOR unit, the system can process massive multiply-accumulate operations on massive amounts of data in one clock cycle, achieving a throughput improvement of approximately 12 times compared to traditional bit-serial solutions.
[0058] Furthermore, those skilled in the art can modify the above architecture in various ways according to actual needs. For example, regarding input precision, although the embodiments demonstrate the processing flow for BF16 format (1 sign bit, 8 exponent bits, 7 mantissa bits), the "comparison tree" and "shift alignment" logic of this invention are also applicable to FP16 (half-precision floating point) or FP32 (single-precision floating point) formats, requiring only corresponding adjustments to the input register bit width and the shifter control logic. Regarding array parallelism, the number of CIM Arrays integrated within the parallel processing unit 10 (32 in the embodiment) can be flexibly adjusted to 16, 64, or other numbers according to throughput requirements and chip area constraints. Such increases or decreases in number do not deviate from the core design concept of "multiple sub-arrays integrated within the PE" of this invention. Regarding alignment strategies, a hybrid alignment scheme can be implemented, i.e., low-overhead layer-wise alignment is used in network layers with small dynamic ranges, while switching to batch-wise alignment via control signals in high dynamic range scenarios such as the Attention layer. Furthermore, the partial summation path and the addition tree can be replaced with a Wallace tree or a compressor array to further reduce the latency and power consumption of the digital logic.
[0059] It is understood that, since the above-mentioned systolic array architecture is used in the in-memory computing architecture of the present invention, the embodiments of the in-memory computing architecture of the present invention include all the technical solutions of all the embodiments of the above-mentioned systolic array architecture, and the technical effects achieved are exactly the same, which will not be repeated here.
[0060] The present invention also proposes a data processing method applicable to the above-mentioned in-memory computing architecture.
[0061] Reference Figure 6 In one embodiment, the data processing method includes the following steps: S100. Define a four-dimensional data format: [CHin+Tin-1) / Tin, H, W, Tin]. The data is arranged in this format, where CH is the original number of channels, H and W are the spatial dimensions of the feature map, and Tin is the input parallelism of the hardware. S200. A dimension folding strategy is adopted for the multi-head self-attention operator, which folds and maps any logical dimension other than the channel dimension to the spatial dimension H.
[0062] In one embodiment, the data processing method further includes the following steps: Before loading, the input data is divided into blocks according to the input parallelism and organized into the shape of [CHinTin, H, W, Tin]. Each clock cycle, a data packet containing Tin elements is read.
[0063] In this embodiment, addressing the challenges of complex operator types (such as convolution, fully connected, and multi-head attention) and variable data dimensions in the visual transformer model, this invention proposes a Data Frame Standardization strategy. Existing accelerators typically require additional buffers and clock cycles to rearrange or transpose data to accommodate different computation layers, leading to pipeline stalls and reduced hardware utilization. This solution uniformly maps all types of tensors (Input / Output, Weights, Q / K / V) in the network to a standardized four-dimensional data packet format: [(CHin+Tin-1) / Tin, H, W, Tin]. By folding and mapping high-dimensional multi-head attention tensors into this standard format, this invention ensures that the data structure output from the previous layer is completely consistent with the input structure required by the next layer. This means that data can flow between network layers with "zero copy," without any hardware or software-level rearrangement operations.
[0064] First, addressing the challenges of diverse operator types (e.g., convolution, matrix multiplication, multi-head self-attention) and varying tensor dimensions in ViT models, this invention implements a unified data flow strategy based on hardware parallelism. To adapt to the fixed input / output parallelism of the hardware array, this invention defines a unified data format based on "channel grouping" (e.g., ...). Figure 1(As shown). The output parallelism factor of the hardware array is set to Tout (e.g., Tout=64 in this embodiment). The system breaks away from the traditional data arrangement based on a single pixel or channel dimension, logically reorganizing all input / output feature map tensors into a four-dimensional standard format: [CHin+Tin] [1Tin, H, W, Tin], where CH is the original number of channels, and H and W are the spatial dimensions of the feature map. For convolution or ordinary matrix multiplication, the data is directly arranged in this format; however, for high-dimensional tensors (usually Head×Pixel×ChannelHead) or any logical dimension other than the channel dimension involved in multi-head self-attention, this invention "folds" them and maps them to the above unified format. Specifically, this embodiment maps the head dimension logic in the multi-head self-attention mechanism to the channel grouping dimension or spatial dimension through dimension folding, or maps any logical dimension other than the channel dimension to the channel grouping dimension or spatial dimension. This mapping makes the high-dimensional tensor consistent with the tensor format of ordinary matrix multiplication in terms of data storage structure, thereby allowing the hardware array to directly reuse the general matrix calculation path to perform operations without adding a dedicated hardware rearrangement circuit. In addition, in view of the physical size limitation (fixed capacity) of the RRAM storage array, this invention implements a matrix block strategy, decomposing large-scale matrix operations into sub-tasks that strictly match the hardware array size (Tin×Tout). Before loading, the input data is divided into blocks according to the input parallelism Tin (Tin=128 in this embodiment), organized into the shape [CHinTin, H, W, Tin]. The hardware reads a data packet containing Tin elements in each clock cycle. At the same time, for the write characteristics of the RRAM array, the original weight matrix (CHout×CHin) is pre-arranged into a series of Tin×Group_Width microblocks. The data structure of each microblock is [128, 64], which directly corresponds to the storage layout of a physical block (e.g., a processing unit group 100) in the RRAM array.
[0065] This invention also proposes a visual model acceleration system.
[0066] In one embodiment, the visual model acceleration system includes the in-memory computing architecture described above.
[0067] In this embodiment, the visual model acceleration system is a hardware-software co-designed RRAM in-memory computing acceleration system. Based on the above embodiments, this visual model acceleration system employs a unified data packaging and task decomposition strategy at the data flow level and a grouped vector systolic array at the hardware architecture level. These two approaches work together to address the mapping mismatch between high-dimensional tensors and two-dimensional RRAM arrays in models such as ViT, and to maximize in-memory computing efficiency.
[0068] Driven by the data strategy in the above embodiments, this invention designs a "grouped vector pulsating array" as the core computing engine. For example... Figure 1 and Figure 5 As shown, this architecture balances the parallelism and timing overhead of RRAM computation through optimized register layout and parallel strategies. For example, the hardware array can contain a total of 64 parallel processing units 10, logically divided into two processing unit groups 100, each containing 32 parallel processing units 10. The specific number of hardware units can be set according to actual conditions and user needs. The input feature vector with a length of Tin=128 is broadcast within the group. Therefore, each processing unit group 100 is responsible for calculating a weight sub-tile with a dimension of 128×32. The two processing unit groups 100 together form a physical output parallelism of Tout=64, enabling the array to achieve an equivalent 128×64 weight computation throughput per cycle under pipeline steady state. In each clock cycle, according to the above unified packaging format, the input activation vector of size Tin (i.e., 128) is broadcast intra-group, that is, directly broadcast to all 32 parallel processing units 10 within the same processing unit group 100. This mechanism makes the behavior within a group similar to a vector processor, leveraging the alignment between the Tin dimension of the input data and the Tin dimension of the weight microblocks to instantly complete a large number of parallel multiplication operations. Simultaneously, the system performs "inter-group pulsation propagation," where input data, or parts thereof, is passed between different PE Groups in a pulsating manner. For example, after the input data completes its computation in the first processing unit group 100, it is passed to the second processing unit group 100 in the next clock cycle. This design maintains full pipeline load while cutting off long-distance broadcast paths and reducing signal fan-out load. To address the issue of high write latency in RRAM devices, this architecture specifically designs a synchronous weight update mechanism. Weight loading is limited to the parallel processing unit level 10, and the system is configured to allow concurrent writing to RRAM macrocells in non-computation rows (or pre-loaded rows of the next layer of weights) while performing computational tasks. This "compute-as-you-go" pipeline mechanism effectively masks the long write latency of RRAM, ensuring the throughput of continuous inference.
[0069] In summary, the technical solution combining "unified data packaging" and "grouped vector pulsation" adopted in this invention brings significant technical effects. Firstly, this solution eliminates the mismatch and rearrangement overhead of high-dimensional tensor mapping. It uses [CHin+Tin]... The unified format [1Tin, H, W, Tin] and the folding of the high-dimensional head dimension of multi-head self-attention into the channel or spatial dimension make the output tensor format structurally mirror the input format required by the next layer. Therefore, when data flows between layers (e.g., from a convolutional layer to an attention layer), it does not require any additional hardware rearrangement units or software-driven memory copies. Experiments show that this design eliminates the data rearrangement bottleneck common in traditional ViT accelerators, ensuring that the grouped vector array is always at peak occupancy without pauses for data alignment. Secondly, this architecture significantly reduces register overhead and chip area. Compared to a full systolic array (10 level-one registers per parallel processing unit), the "grouped vector" design drastically reduces the number of registers, directly resulting in higher computational density within a limited edge device area budget. Furthermore, this scheme improves throughput and optimizes timing convergence. Combining the high parallelism of vector broadcasting and the short-path characteristics of systolic arrays, this architecture requires only Tout+1 cycles to complete the computation for a weight block of size Tin×Tout (where Tout refers to the output dimension of the block), significantly fewer than the 2Tin+Tout–1 required by traditional systolic arrays and the Tin+Tout–1 required by vector arrays. Simultaneously, the grouping design avoids the huge fan-out latency caused by global broadcasting, enabling the system to operate at higher frequencies and achieving a 2.8x improvement in energy efficiency. Finally, this scheme maximizes the utilization of the RRAM array. By pre-arranging the weight matrix into micro-blocks matching the physical size of the PEGroup (128×64), RRAM cell padding caused by irregular matrix shapes is avoided. This invention ensures that each computation is performed on an effective micro-block, thereby maintaining a high computational density of the RRAM array when processing matrix multiplications of different dimensions.
[0070] Regarding the grouping size, the embodiment of this solution demonstrates a configuration of Tin=128, Tout=64 (i.e., 2 processing unit groups 100, each with 32 parallel processing units 10). However, in practical applications, the number of processing unit groups 100 (e.g., 4 groups, 8 groups) and the number of parallel processing units 10 within each group can be flexibly adjusted according to the specific neural network model size and hardware resource constraints. For example, for shallow networks with fewer channels, the packing granularity can be dynamically adjusted to reduce invalid padding. Regarding the storage medium, although this embodiment mainly describes RRAM, this architecture is also applicable to other non-volatile memories with similar storage and computation characteristics, such as magnetoresistive random access memory, phase-change memory, or ferroelectric field-effect transistors. As long as the storage cells support matrix arrangement and in-situ computation, the unified packing and grouping ripple control logic of this invention can be used. Regarding precision and sparsity support, while primarily describing the processing of BF16 or binarized weights, this array architecture can extend its support for in-memory computations of INT4, INT8, or other mixed precisions by integrating bit-serial logic or multi-bit ADCs within the parallel processing unit 10. Simultaneously, based on a unified packet format, bitmasks or index vectors can be introduced, storing only non-zero "groups" in the data packets and indicating the location of valid data through additional index metadata, thereby further compressing storage space and reducing power consumption from invalid RRAM flips. Regarding data flow direction... Figure 5 As a variation of the architecture, which activates broadcasting and pulsation for input, the architecture can also be configured to output a fixed stream, where the input data flows completely through the array while partial sums are accumulated within the parallel processing unit 10 until completion; or the directions of broadcasting and pulsation can be interchanged (e.g., broadcasting in the column direction and pulsating in the row direction) to accommodate different physical layout constraints.
[0071] It is understood that, since the above-mentioned in-memory computing architecture is used in the visual model acceleration system of the present invention, the embodiments of the visual model acceleration system of the present invention include all the technical solutions of all the embodiments of the above-mentioned in-memory computing architecture, and the technical effects achieved are exactly the same, which will not be repeated here.
[0072] In one embodiment, the visual model acceleration system further includes: The controller stores a data processing program, which, when executed, performs the data processing method described above on the in-memory computing architecture.
[0073] In this embodiment, the controller can store data processing programs. When executing these programs, the controller reads and writes data according to a unified format of [(CHin+Tin-1) / Tin, H, W, Tin], and folds the header dimension to the spatial dimension H. This allows the computing array to reuse the same set of group vector systolic control logic when processing convolutional, fully connected, and attention layers. This coupled design eliminates the need for mode switching or increased latency.
[0074] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A pulsating array architecture, characterized in that, include: Multiple processing unit groups, each of the processing unit groups including multiple parallel processing units; Each of the processing unit groups integrates a memory unit for storing weights, and the processing unit groups adopt a weight-resident mode. Multiple parallel processing units are used to perform calculations on the input data according to weights; Within each clock cycle, each processing unit group broadcasts the input activation vector directly to multiple parallel processing units within each processing unit group according to a unified packet format. Among the multiple processing unit groups, the input activation data is passed step by step in a pulsating manner.
2. The pulsating array architecture as described in claim 1, characterized in that, The parallel processing unit includes: An XOR array, configured with XOR logic circuitry, is used to perform XOR logic operations on input data according to weights to generate multi-way partial products. The addition tree module is used to perform spatial dimension reduction and summation on the multi-way partial products generated by logic circuit operations.
3. The pulsating array architecture as described in claim 1, characterized in that, The memory unit is one of resistive random access memory, magnetoresistive random access memory, phase change memory, or ferroelectric field-effect transistor.
4. An in-memory computing architecture, characterized in that, It includes an input module, an output module, and a pulsating array architecture as described in any one of claims 1-3, wherein the input module is used to output raw data to the pulsating array architecture, and the output module is used to output data output by the pulsating array architecture to an external module.
5. The in-memory computing architecture as described in claim 4, characterized in that, The input module includes: An asynchronous first-in-first-out queue is used to output raw data from external modules after bandwidth matching. An input buffer is used to cache the data output by the asynchronous first-in-first-out queue; The activation value conversion module is used to parse and convert the floating-point format data output from the input buffer into a tail value. The activation value pulsation control module performs vectorized recombination and distribution of the bit values output by the activation value conversion module based on the group vector pulsation strategy.
6. The in-memory computing architecture as described in claim 4, characterized in that, The output module includes: A shift adder is used to shift and add the data output by the pulsating array architecture before outputting it. An accumulator is used to accumulate the data output from the shift adder and then output the result. The output buffer is used to cache the data output by the accumulator for external modules to read.
7. A data processing method applied to the in-memory computing architecture according to any one of claims 4-6, characterized in that, Includes the following steps: Define a four-dimensional data format: [CHin+Tin-1) / Tin, H, W, Tin], where the data is arranged in this format, where CH is the original number of channels, H and W are the spatial dimensions of the feature map, and Tin is the input parallelism of the hardware. A dimension folding strategy is adopted for the multi-head self-attention operator, which folds and maps any logical dimension other than the channel dimension to the spatial dimension H.
8. The data processing method as described in claim 7, characterized in that, It also includes the following steps: Before loading, the input data is divided into blocks according to the input parallelism and organized into the shape of [CHinTin, H, W, Tin]. Each clock cycle, a data packet containing Tin elements is read.
9. A visual model acceleration system, characterized in that, Including the in-memory computing architecture as described in any one of claims 4-6.
10. The visual model acceleration system as described in claim 9, characterized in that, Also includes: A controller stores a data processing program, which, when executed, performs the data processing method as described in any one of claims 7-8 on the in-memory computing architecture.