In-memory computing architecture for hybrid AI load
By dividing the in-memory computing architecture into static and dynamic page regions, the inefficiency of hybrid AI workloads in existing technologies is solved, achieving efficient data reading and computing capabilities and improving the processing performance of hybrid AI workloads.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SEMICON TECH INNOVATION CENT(BEIJING) CORP
- Filing Date
- 2026-01-14
- Publication Date
- 2026-04-28
AI Technical Summary
Existing in-memory computing architectures cannot efficiently handle both memory-intensive and compute-intensive tasks simultaneously when dealing with mixed AI workloads, resulting in performance bottlenecks and resource waste.
It adopts a heterogeneous in-memory computing architecture, dividing the space into static page regions and dynamic page regions for memory-intensive and computation-intensive tasks, respectively. It integrates an instruction decoder to coordinate the work and supports efficient matrix transpose and sparse computation.
Without increasing area and power consumption, it improves the data reading capability of memory-intensive tasks and the flexibility of computation-intensive tasks, thereby enhancing the processing efficiency of mixed AI workloads.
Smart Images
Figure CN121935207A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of in-memory computing technology, and in particular relates to an in-memory computing architecture for hybrid AI workloads. Background Technology
[0002] With the rapid development of artificial intelligence technology, modern AI applications, especially complex models such as automatic speech recognition and large language model systems, pose unprecedented challenges to the underlying hardware. These advanced AI workloads typically contain multiple computational modes with distinct characteristics, exhibiting significant hybrid features. These workloads simultaneously possess both memory-intensive and compute-intensive computational layers. For memory-intensive computational layers, the number of weight parameters that need to be stored is relatively small, but they are accessed and read extremely frequently during computation. This high-frequency, repetitive memory access operation puts enormous pressure on the bandwidth and power consumption of the storage subsystem, a typical "memory wall" bottleneck problem. Computationally intensive computational layers (such as Transformer-based attention mechanisms) are characterized by complex computational logic and variable, irregular data flows. They not only involve a large number of matrix operations but also often involve special data operations, such as matrix transpose and sparse computation, which require computational units with high flexibility and data processing capabilities.
[0003] To address the "memory wall" problem in AI computing, in-memory computing (CIM) technology emerged. However, most current mainstream CIM architectures adopt a typical weighted static, input-broadcast structure. This means that the storage and computing units within the entire CIM macrocell use a uniform, identical structure. This homogeneous architecture can typically only be optimized for a single type of computing task during design. When faced with the aforementioned mixed AI workloads that are both memory-intensive and computationally intensive, its limitations become glaringly apparent. These limitations include:
[0004] 1. Insufficient optimization for memory-intensive tasks: If designed for flexibility, its storage cell density and read efficiency may not meet the needs of high-frequency weighted access, resulting in performance bottlenecks.
[0005] 2. Lack of flexibility for computationally intensive tasks: If designed for high-density storage, its internal data paths and computational logic are often too simple, making it unable to efficiently perform complex operations such as matrix transpose or sparse computation.
[0006] Therefore, traditional homogeneous CIM architectures cannot efficiently handle these two distinct computing challenges simultaneously when dealing with mixed AI workloads, ultimately leading to unnecessary waste and compromises in chip area, system power consumption, and end-to-end performance. Summary of the Invention
[0007] The purpose of this invention is to provide an in-memory computing architecture for hybrid AI workloads, in order to solve the problems existing in the prior art.
[0008] To achieve the above objectives, the present invention provides an in-memory computing architecture for hybrid AI workloads, comprising:
[0009] A static page area is used for memory-intensive tasks to achieve permanent storage and retrieval of weights;
[0010] A dynamic page region, which is used for computationally intensive tasks and supports matrix transpose and array sparse computation;
[0011] The peripheral circuit includes an instruction decoder, which is shared by the static page region and the dynamic page region. The instruction decoder is used to parse external instructions and generate control signals to coordinate the operation of each region.
[0012] Optionally, the peripheral circuitry further includes:
[0013] A read / write driving circuit is used to convert control signals into actual driving current / voltage to control the read / write operations of the static page area.
[0014] Optionally, the static page area specifically includes:
[0015] A static storage array, wherein the static storage array is used to hold weight parameters;
[0016] A local decoder, which converts control signals into micro-operation signals that are locally executable by the static storage array;
[0017] Sub-line driver, which is used to generate drive signals for rows / columns within the static storage array to control the read / write state.
[0018] Optionally, the dynamic page area specifically includes:
[0019] A two-dimensional writable unit that supports flash memory input to achieve efficient sparse computation;
[0020] A vertical input channel, located inside a two-dimensional writable cell, is used to input data in parallel into multiple columns of the storage array, thereby achieving in-memory matrix transposition.
[0021] Optionally, when performing sparse computation, columns containing non-zero elements are activated and receive valid input data, while columns containing zero elements are skipped.
[0022] The technical effects of this invention are as follows:
[0023] This invention provides a design for a compute-in-memory (CIM) macrocell to accelerate artificial intelligence (AI) workloads. The design employs a heterogeneous architecture and is designed to efficiently handle modern AI applications with hybrid computing modes.
[0024] This invention addresses the inefficiency of existing in-memory computing (CIM) architectures when handling mixed-mode AI workloads through an integrated CIM architecture. This invention provides high-throughput data reading capabilities for memory-intensive tasks and highly flexible data processing and computing capabilities for compute-intensive tasks without sacrificing area and power efficiency. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0027] Figure 1 This is a schematic diagram of the overall structure of the in-memory computing architecture for hybrid AI workloads in an embodiment of the present invention;
[0028] Figure 2 This is a schematic diagram of the structure of the static page area in an embodiment of the present invention;
[0029] Figure 3 This is a schematic diagram of the structure of the dynamic page area in an embodiment of the present invention; Detailed Implementation
[0030] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.
[0031] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.
[0032] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.
[0033] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.
[0034] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0035] like Figure 1 - Figure 3 As shown, this embodiment provides an in-memory computing architecture for hybrid AI workloads, including:
[0036] A static page area is used for memory-intensive tasks to store and retrieve weights.
[0037] A dynamic page region, which is used for computationally intensive tasks and supports matrix transpose and array sparse computation;
[0038] The peripheral circuit includes an instruction decoder, which is shared by the static page region and the dynamic page region. The instruction decoder is used to parse external instructions and generate control signals to coordinate the operation of each region.
[0039] This embodiment proposes an in-memory computing macrostructure. Its core idea is to divide and integrate two functional regions that are deeply optimized for different computing tasks within a single CIM macro unit, and to support matrix transpose and array sparse computing.
[0040] The overall architecture of the in-memory computing macrostructure provided in this embodiment is as follows: Figure 1 As shown, the in-memory computing macrounit in this embodiment is physically a single unit, but logically divided into two heterogeneous partitions:
[0041] High-density static pages area 101: This area is specifically designed for memory-intensive tasks.
[0042] 2D-Writable Dynamic Pages Area 102: This area is specifically designed for computationally intensive tasks.
[0043] The high-density static page region 101 and the two-dimensional writable dynamic page region 102 share some peripheral circuits, such as instruction decoders, thereby achieving higher integration and resource utilization than using two independent macros.
[0044] like Figure 2 As shown, the design goal of static page area 101 is to achieve permanent storage and most efficient retrieval of weights. Its key features include:
[0045] Storage unit 201: Utilizes SRAM storage units to fix the weight parameters, reducing power consumption during data transfer. This is crucial for storing weights that remain constant throughout the model inference process. For example... Figure 2 Since the weights in this region are "static" (i.e., write once, read many times), its peripheral circuitry includes a local decoder and precharge and input / output circuitry.
[0046] like Figure 3 As shown, the design goal of the dynamic page area 102 is to provide extremely high flexibility for tasks with variable data flows and complex calculations. Its core is an innovative 2D-Writable Cell (DLC) and its unique associated data pathway. This design achieves the following key functions:
[0047] 1. Supports column-wise write to achieve efficient in-memory matrix transposition. Figure 3 (Top right part)
[0048] Traditional SRAM arrays typically only support row-wise writes. This embodiment adds a vertical input channel (floating-point input preprocessing) and corresponding control logic to the dynamic page area, such as... Figure 3 As shown in the upper right, when performing a matrix transpose operation, data can be written row-wise from the vertical input channel to multiple columns of the storage array; this allows time-consuming data rearrangement operations that originally required an external processor to be completed efficiently directly within the CIM macrocell, greatly improving the performance of tasks such as Transformer attention mechanisms.
[0049] 2. Supports flash input for efficient sparse computing. Figure 3 ):
[0050] The dynamic page area in this embodiment also includes a "flash input" mechanism. For example... Figure 3 As shown in the lower left section, this mechanism allows for the configuration of unique input data for specific columns in the storage array through dedicated column selection logic. This capability enables the architecture to natively support the computation of sparse weights. When processing a sparse matrix, only columns containing non-zero elements are activated and receive valid input data, while columns containing zero elements are skipped, thus avoiding a large number of invalid multiplication operations and significantly improving the efficiency and energy efficiency of sparse computation.
[0051] In summary, this embodiment solves the problem of low efficiency of existing in-memory computing architectures when handling mixed-mode AI workloads through the integrated CIM architecture. It can provide high-throughput data reading capabilities for memory-intensive tasks and highly flexible data processing and computing capabilities for compute-intensive tasks without sacrificing area and power efficiency.
[0052] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. An in-memory computing architecture for hybrid AI workloads, characterized in that, include: A static page area is used for memory-intensive tasks to store and retrieve weights. A dynamic page region, which is used for computationally intensive tasks and supports matrix transpose and array sparse computation; The peripheral circuit includes an instruction decoder, which is shared by the static page region and the dynamic page region. The instruction decoder is used to parse external instructions and generate control signals to coordinate the operation of each region.
2. The in-memory computing architecture for hybrid AI workloads according to claim 1, characterized in that, The peripheral circuit also includes: A read / write driving circuit is used to convert control signals into actual driving current / voltage to control the read / write operations of the static page area.
3. The in-memory computing architecture for hybrid AI workloads according to claim 1, characterized in that, The static page area specifically includes: A static storage array, wherein the static storage array is used to hold weight parameters; A local decoder, which converts control signals into micro-operation signals that are locally executable by the static storage array; Sub-line driver, which is used to generate drive signals for rows / columns within the static storage array to control the read / write state.
4. The in-memory computing architecture for hybrid AI workloads according to claim 1, characterized in that, The dynamic page area specifically includes: A two-dimensional writable unit that supports flash memory input to achieve efficient sparse computation; A vertical input channel, located inside a two-dimensional writable cell, is used to input data in parallel into multiple columns of the storage array, thereby achieving in-memory matrix transposition.
5. The in-memory computing architecture for hybrid AI workloads according to claim 4, characterized in that, When performing sparse computation, columns containing non-zero elements are activated and receive valid input data, while columns containing zero elements are skipped.