A design method of a storage location-aware programming system for a RISC-V storage-computing integrated chip

Abstract

The application provides a design method of a storage location-aware programming system for a RISC-V storage-computing integrated chip, and belongs to the technical fields of computer programming systems and heterogeneous computing chips. The method writes a CIM computing core function through a decorator and type annotation provided by a Python DSL, explicitly declares a physical storage location of data in a multi-level storage hierarchy of a CIM chip through storage location type annotation, performs storage location legality checking, and automatically inserts a data carrying primitive when there is inconsistency; a runtime environment models the CIM computation as an external call to the CPU, forms a three-step collaborative execution semantics of "dispatch task-backfill result-uninstall dependency" for the CIM computation, and automatically manages the parallel scheduling of the CIM computation; the runtime environment controls the granularity of the dispatched task based on a heartbeat mechanism, and achieves the effects of short-term CIM computation synchronization and long-term CIM computation asynchrony through a configurable integral value. The programming system designed by the application is suitable for software development in AI model reasoning, edge computing and other scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer programming systems and heterogeneous computing chip technology, specifically to a design method for a storage location-aware programming system for RISC-V in-memory computing chips. Background Technology

[0002] The rapid development of artificial intelligence technology continues to place new demands on hardware computing power and energy efficiency. Computing in Memory (CIM) chips overcome the energy consumption and latency bottlenecks of data movement in the traditional von Neumann architecture by directly executing core operators such as matrix multiplication within the memory array. However, CIM chips often employ dedicated designs, introducing severe system fragmentation problems, and the instruction control and data movement between the CPU and CIM chip significantly impact energy consumption and latency.

[0003] The RISC-V instruction set architecture offers a solution to this system fragmentation problem. Originating as an educational foundation, RISC-V has evolved into a globally approved and scalable standard, providing a legally unhindered and highly customizable basis for academic research and commercial silicon integration. RISC-V's open-source nature and flexible support for instruction extensions enable tight monolithic integration of scalar CPUs, vector coprocessors, and matrix execution engines within a unified architectural framework, making it the mainstream architecture for CIM chips. However, to fully utilize the theoretical peak performance of such heterogeneous RISC-V systems, the programming systems used by developers need to be redesigned, allowing them to optimize for specific CIM chip characteristics while preserving high-level algorithmic abstractions.

[0004] Based on the characteristics of on-chip storage, programming systems suitable for CIM need to carefully consider storage configuration. A typical CIM chip involves multiple storage levels: (1) Global Storage RAM (PSRAM): large-capacity off-chip storage with limited bandwidth; (2) On-chip Scratch Pad (SPAD): multiple on-chip SRAMs, providing high-bandwidth, low-latency local data cache; (3) CIM Clusters: core computing units that couple storage and computation, organized at the page level, supporting in-situ matrix operations. Based on the characteristics of instruction control, RISC-V-based CIM chips often adopt an extended instruction set to directly drive the CIM chip, while instructions unrelated to on-chip computation are directly processed by the CPU, which places demands on the scheduling of heterogeneous architectures.

[0005] Existing programming systems for CIM chips suffer from three main problems. First, the storage location is opaque: developers need to manually manage data transfer between PSRAM, SPAD, and CIM via low-level APIs, lacking abstraction at the programming language level. The location of data within the three-tiered storage hierarchy is crucial to program correctness and performance. Existing systems often directly call inline assembly or intrinsic interfaces, failing to incorporate storage location into the programming language design. Second, parallelism management is manual: for example, multiple pages in the CIM array can perform matrix calculations in parallel, and multiple blocks in the SPAD can perform data prefetching in parallel. However, in existing systems, this parallelism relies entirely on manual arrangement by developers, often requiring explicit insertion of synchronization and locking primitives in the code and manual management of CIM page resource allocation. Third, scheduling granularity control is difficult: excessive parallelism introduces scheduling overhead, such as creating CPU threads and waiting for CIM calculations to return. However, insufficient parallelism fails to fully utilize hardware resources, such as the parallel execution of different types of computational tasks on the CPU and CIM chip. Existing systems lack automated management of this scheduling granularity, often requiring developers to manually adjust it for specific problems.

[0006] In the field of artificial intelligence, programming systems for heterogeneous computing mainly fall into two categories. One category consists of user-schedulable languages, such as Halide, Tiramisu, and Exo. Their core characteristic is the strict division of the program into two components: algorithm and scheduling. The algorithm defines mathematical operations within a purely functional context, unaffected by implementation details, while scheduling defines the execution mechanism, such as memory layout, parallelization strategies, and hardware accelerator offloading. However, these programming systems do not address practical scheduling problems, such as the runtime scheduling method adapted to RISC-V in-memory computing chips considered in this invention. The other category consists of tile-based domain-specific languages ​​(DSLs), such as TileLang, cuTile, and Tile Language Ascend. Their core characteristic is the abstraction of tensor operations from the GPU's single-instruction multithreaded (SIMT) logic through tile partitioning, simplifying operator implementation and heterogeneous scheduling. However, these programming systems are primarily designed for GPUs or dedicated accelerators and do not consider the characteristics of the RISC-V in-memory computing architecture involved in this invention. Therefore, it is of great significance to study a design method for a memory location-aware programming system for RISC-V in-memory computing chips. Summary of the Invention

[0007] To address the shortcomings of existing programming systems in supporting the RISC-V in-memory computing architecture, particularly the issues of opaque storage locations, manual parallelism management, and difficulties in managing scheduling granularity, this invention proposes a design method for a storage location-aware programming system for RISC-V in-memory computing (CIM) chips. This method is implemented using a domain-specific language (Python DSL) built with Python syntax. It declares the storage locations of data such as vectors, matrices, and tensors through type annotations, analyzes the consistency between the storage locations and the requirements of CIM operators through type checking, and supports the automatic insertion of data transfer primitives when inconsistencies occur. The runtime environment of this programming system automatically manages the parallel scheduling of CIM computations, managing both the collaboration between the CPU and the CIM chip, as well as the parallel granularity between computational tasks.

[0008] The technical solution of the present invention is as follows:

[0009] A design method for a storage location-aware programming system for RISC-V in-memory computing (CIM) chips, characterized in that the method includes the following steps:

[0010] (1) Design a storage location-aware type system and Python DSL;

[0011] The physical storage location of data in a RISC-V Compute-in-Memory (CIM) chip is elevated to a first-class attribute in the programming language type system. Developers write CIM computation core functions in the Python DSL frontend using decorators and type annotations provided by Python DSL. These type annotations include storage location type annotations, which are used to explicitly declare the physical storage location of data in the multi-level storage hierarchy of the CIM chip. Based on the three-level storage hierarchy involved in the CIM chip, the values ​​of the storage location type annotations include PSRAM, SPAD, or CIM. In the storage location-aware type system, constraints on the storage location are defined for each type of CIM computation.

[0012] The program statements of the CIM calculation core function are traversed to extract the storage locations of operands related to CIM calculation. These locations are compared with predefined constraint rules. If the current storage location of the operand satisfies the constraint rules, the type system inserts transport primitives as needed and constructs a data flow graph. If the constraint rules are not satisfied, the type system automatically transports and inserts data according to preset transport rules and then constructs a data flow graph. Finally, the data flow graph is passed to the runtime environment. Each node in the data flow graph represents a statement, and each directed edge represents a data dependency.

[0013] (2) Design an automatic collaborative method for CIM calculation and CPU calculation;

[0014] Design a runtime environment adapted to the aforementioned Python DSL. This runtime environment divides computation into CPU computation and CIM computation. CPU computation is directly executed by a RISC-V based CPU core, while CIM computation is initiated by the CPU through extended instructions, but the actual execution occurs in the CIM array or Direct Memory Access (DMA) engine. The runtime environment then interprets and evaluates functions marked with specific decorators in the Python DSL, including the following three steps:

[0015] Step 1: Dispatch the task; When the runtime environment executes a CIM statement, it encapsulates the statement into a CIM task and submits it to the CIM task execution queue. At the same time, a task placeholder is set for the result variable of the statement in the program's execution state. The CPU then continues to check and execute subsequent statements that do not depend on the task marker.

[0016] Step 2: Backfilling results; After the CIM hardware completes a task, it notifies the runtime environment through hardware interrupt or polling mechanism. The runtime environment replaces the corresponding task placeholder with the actual result tensor generated by CIM calculation, updates the storage location information of the result tensor, and marks the task as "completed".

[0017] Step 3: De-dependency; After backfilling the results, the runtime environment rescans the queue of statements to be executed to check if any statements that were previously blocked due to their dependence on the results of this task have become executable. The check is performed by decrementing the "unresolved dependency" counter of the statements that depend on this task. If the counter reaches zero, the statements are marked as "steppable". If there are "steppable" statements, they are added back to the executable set and executed in the next interpretation and evaluation step.

[0018] When the runtime environment interprets and evaluates functions marked with specific decorators in the Python DSL, it simultaneously maintains a statement execution state table, recording the state of each statement in the program: "Not Ready", "Steppable", and "Completed". It also maintains a "Unresolved Dependencies" counter based on the data flow graph for each statement. When one of its predecessor nodes completes and backfills its result, the counter for that statement is decremented by one. When it reaches zero, it indicates that it has entered the "Steppable" state. The runtime environment executes the following decision process in each interpretation and evaluation step:

[0019] First, iterate through the incomplete statements and check if each statement has been parsed into actual data. If so, mark it as "steppable". Then, in the same evaluation step, execute all "steppable" statements. For CIM calculation type statements, perform the "dispatch task" action, that is, send the statement to the CIM hardware and set the result as a placeholder. For CPU calculation type statements, directly interpret and evaluate them. After completing one round of execution, determine if there are any incomplete statements. If so, proceed to the next round of interpretation and evaluation.

[0020] (3) Design an automatic resource scheduling method between computational tasks;

[0021] This automatic resource scheduling method, in conjunction with the automatic coordination method between CIM computing and CPU computing, is used to optimize the parallel granularity between computing tasks;

[0022] When interpreting and evaluating functions marked with specific decorators in the Python DSL, the runtime environment automatically marks all CIM computations as "potentially parallel," performing them serially and synchronously by default. After the CPU issues a CIM instruction, it waits for the result in place. At the same time, the runtime environment maintains three different versions of the subsequent execution path in the call stack frame: the first is the serial version, which is the default execution path; the second is the parallel version, which is the execution path of the current CPU thread after the result placeholder of the CIM task when the runtime scheduler decides to make the CIM computation asynchronous; and the third is the CIM version, which represents the execution path that waits for the asynchronous CIM computation to complete and backfill the result.

[0023] The runtime environment maintains a global integral manager, implementing a "heartbeat" through a timer interrupt in the RISC-V core. The interrupt interval N and the integral C obtained with each heartbeat are configured. Every N microseconds, the runtime environment distributes C integrals to each active CPU thread via an interrupt handler. These integrals are recorded in each thread's local state. When a thread's accumulated integrals reach a certain threshold, the runtime environment selects the thread with the longest accumulated integrals, consumes the threshold number of integrals, and creates a CIM version execution path to wait for the CIM calculation to complete and backfill the result. The current thread marks the result as a placeholder and switches the code execution path to the concurrent version, continuing to concurrently execute dependent statements. When the asynchronously executed CIM task is completed, the runtime environment notifies the original thread via a synchronization primitive or interrupt mechanism. The CIM version execution path then writes back the result and clears the placeholder, assisting the runtime environment in de-depending on other threads.

[0024] The design method of the memory location-aware programming system for RISC-V in-memory computing chips of the present invention has the following beneficial effects:

[0025] (1) The programming system is aware of the storage location: The Python DSL designed by the method of this invention encodes the three-level storage hierarchy (PSRAM, SPAD, CIM) of the CIM chip as part of the vector, matrix and tensor data types, so that the type system can statically perform storage location legality checks and automatic optimization, which improves the level of abstraction for programming RISC-V in-memory computing chips, and allows developers to fully consider the physical characteristics of the CIM chip when programming.

[0026] (2) CIM parallel scheduling transparent to developers: The runtime environment designed by the method of this invention models CIM calculation as an external call to the CPU. Thus, through the three-step execution process of "dispatch task - backfill result - remove dependency", the semantics of CIM calculation and CPU calculation are separated according to the storage location information, so that the CPU does not need to wait for CIM calculation to complete, and at the same time, developers do not need to manually insert synchronization primitives, reducing the programming burden of developers.

[0027] (3) Low-overhead automatic scheduling of computing tasks: The runtime environment designed by the method of the present invention further controls the granularity of "dispatch tasks" based on the "heartbeat" mechanism, avoiding the creation of too many CPU threads and causing a large amount of thread management overhead. The effect of short-term CIM calculation is achieved by configurable integral values, while not requiring developers to manually control the granularity. Furthermore, performance can be optimized according to the specific CIM chip hardware.

[0028] (4) The programming system designed by the method of the present invention is suitable for software development in scenarios such as AI model inference and edge computing. It allows developers to make full use of the in-memory computing characteristics of RISC-V in-memory computing chips during the programming stage, efficiently realize data transfer, in-memory computing, task scheduling and other links, and fully tap the potential of hardware computing power. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 The diagram shows the programming front-end and type system designed for the design method of the storage location-aware programming system for RISC-V in-memory computing chips proposed in this invention. It illustrates the type annotation extension of Python DSL and the storage location-aware checking and optimization process performed by the type system.

[0031] Figure 2This diagram illustrates the runtime environment of the design method for the storage location-aware programming system for RISC-V in-memory computing chips proposed in this invention. It shows the flow of the automatic collaboration method between the CPU and the CIM chip, as well as the automatic resource scheduling method between computing tasks. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0033] This invention relates to a three-level storage hierarchy for RISC-V in-memory computing heterogeneous chips. The RISC-V-based CPU core is responsible for control and scalar computation, driving the CIM computing array and Direct Memory Access (DMA) through extended instructions. The three storage levels are: Global Storage RAM (PSRAM), which has a large capacity but high access latency and can be used to store complete model parameters and large-scale intermediate results; On-chip Spadden Memory Area (SPAD), composed of multiple independent on-chip SRAMs, providing high-bandwidth, low-latency data caching, mainly used to store activation values, input / output buffers, and intermediate data required for the current computation; and the CIM computing array, composed of multiple CIM clusters, each containing several computation pages, which is a unit coupling storage and computation, capable of performing matrix multiplication and other operations in-situ within the storage array.

[0034] The three-level storage is connected by two main data paths: the first is a DMA transfer path between PSRAM and SPAD, used to transfer data between global storage and on-chip temporary storage; the second is a weight loading path from SPAD to CIM array, used to write weight data from the on-chip temporary storage area SPAD to a designated page of CIM array; and the third is a result output path from CIM array to SPAD, used to write the result to SPAD used for output buffer after the CIM array has completed the calculation.

[0035] The programming system designed in this invention serves as a software layer overlaying a RISC-V in-memory computing heterogeneous chip architecture, including a Python DSL front-end, a storage location-aware type system, and a runtime environment (runtime cooperative scheduling engine). The design method of this invention includes the following steps:

[0036] (1) Design a storage location-aware type system and Python DSL

[0037] The core design concept of this invention is to elevate the physical storage location of data on a RISC-V in-memory computing chip to a first-class attribute within the programming language type system. For example... Figure 1As shown, the Python DSL front-end designed in this invention allows developers to write CIM calculation core functions through decorators and type annotations. This Python DSL defines a parameterized type Tensor[Location, DataType, Shape], unifying vector, matrix, and tensor types through the Shape parameter. Compared to traditional type annotations, a storage location type annotation Location is added, which can take values ​​such as PSRAM, SPAD, and CIM. Other DataType parameters (such as INT4, INT8, INT32, FP8, etc.) and Shape parameters (such as dimension and size) are consistent with traditional tensor type annotations. The PSRAM type indicates that data is stored in global storage, i.e., large-capacity off-chip storage, but with limited bandwidth. This type carries the base address information and capacity constraints of global storage. Tensors annotated as PSRAM type are usually complete model weights or large-scale data not yet loaded onto the chip. The SPAD type indicates that data is located in the on-chip temporary storage area, characterized by high bandwidth and low latency, and has multiple independent SRAMs. Since the on-chip temporary storage area consists of multiple independent SRAMs, the SPAD type is further refined into numbered subtypes (such as SPAD_0, SPAD_1) according to the hardware design. Each subtype corresponds to an independent on-chip SRAM unit, carrying the unit's base address and capacity information. When developers do not specify a specific number, the type system will automatically allocate SPAD units. CIM type representations indicate that data resides in the CIM computing array, achieving storage and computation coupling, such as in-situ matrix operations. This can be further refined according to hardware design (e.g., refined to the CIM page level for finer-grained storage management: page 0, page 1, page 2, etc.). Once weight data is written to a CIM page, it can participate in matrix multiplication operations in-situ without additional data transfer. These three storage location types constitute a hierarchical type system. This system not only describes the logical ownership of data but also implicitly contains physical constraints of the hardware, providing type-level support for subsequent validity checks and automatic data transfer optimization. In Python DSL, the physical storage location is explicitly declared through type annotations, for example... Figure 1 The above is given:

[0038] weight: Tensor[CIM, "int8", (128, 128)] # Weights reside in the CIM array

[0039] input: Tensor[SPAD, "int8", (1, 128)] # Input is located in ScratchPad

[0040] output: Tensor[SPAD, "int32", (1, 128)] # Output is located in ScratchPad

[0041] model: Tensor[PSRAM, "int8", (512, 512)] # Model parameters are located in global storage

[0042] In the RISC-V in-memory computing architecture, different CIM calculations have certain physical requirements for the storage location of operands. This invention supports defining storage location constraints for each type of CIM calculation in the type system, and performing static storage location validity checks through type annotation parsing, for example (depending on specific hardware requirements):

[0043] (1) Matrix multiplication (GEMM) may require the weight matrix to be located in the CIM array (i.e., labeled as CIM type), the activation vector or matrix to be located in the on-chip temporary storage area (i.e., labeled as SPAD type), and the calculation result to be output to the on-chip temporary storage area (i.e., labeled as SPAD type). Convolution and other operations have similar requirements, and their inputs and outputs are both labeled as SPAD type.

[0044] (2) Vector operations (such as element-wise addition, ReLU activation, etc.) may require all operands and outputs to be located in the on-chip temporary storage area SPAD.

[0045] (3) Data transfer itself also has storage location constraints. For example, the source of DMA loading must be PSRAM and the destination must be SPAD; the DMA storage direction is opposite; the source of CIM writing must be SPAD and the destination must be CIM.

[0046] During type checking, each statement in the CIM calculation core function is sequentially traversed, and the storage location type parameters of each operand are extracted and compared with the constraint rules corresponding to that statement. If the current storage location of the operand meets the constraint rules, the check passes; otherwise, it is reported to the developers and an option to automatically insert data transfer primitives is provided.

[0047] If the developers decide that the programming system will automatically handle data transfer and insertion, the type system will follow the following transfer rules:

[0048] (1) Scenario 1: The operand is located in PSRAM and needs to be used in SPAD. The type system first allocates a buffer of the same size in SPAD for the tensor, then inserts a DMA load instruction to move the data from PSRAM to the allocated SPAD buffer, and finally updates the reference of the operand in the statement to the SPAD buffer.

[0049] (2) Scenario 2: The operand is located in SPAD and needs to be used in the CIM array. The type system allocates enough CIM pages for the tensor in the CIM array, then inserts a CIM write instruction to write the data from SPAD to the CIM pages, and finally updates the operand reference.

[0050] (3) Scenario 3: The operand is located in PSRAM and needs to be used in the CIM array. Since there is no direct path between PSRAM and CIM, the system needs to perform two steps of data transfer: first, insert a DMA load instruction to transfer the data from PSRAM to SPAD, and then insert a CIM write instruction to load the data from SPAD to the CIM page.

[0051] When allocating SPAD buffers and CIM pages, the type system maintains a resource tracking table based on hardware configuration information, recording currently occupied SPAD units and CIM pages, and manages them according to user-configured allocation strategies. When resources are insufficient, the type system performs a lightweight lifecycle analysis and reclaims unused SPAD buffers or CIM pages.

[0052] Through the above checks and optimizations, the Python DSL program is converted into a data flow graph and an optimized instruction sequence. The conversion process is as follows: First, the program statements are traversed to extract the storage locations of operands related to CIM calculation. Then, these locations are compared with predefined constraint rules. If the current storage location of the operand satisfies the constraint rules, transport primitives are inserted as needed to construct a data flow graph (nodes represent operation statements, and directed edges represent data dependencies). Finally, the graph is passed to the runtime environment.

[0053] This invention's Python DSL is implemented using two native Python syntax mechanisms: decorators and type annotations, without requiring modification to the Python interpreter. Developers use the `@cim_kernel` decorator to mark a Python function as a core function for CIM calculation; functions marked with this decorator receive special handling from the type system. In the function signature, developers use storage location-aware type annotations to label tensors, declaring their storage location, data type, and shape. Within the function body, developers write computational logic in a near-mathematical expression style, calling native operators (such as GEMM) provided by the RISC-V in-memory computing chip modeled in the programming system, without needing to concern themselves with the underlying data handling.

[0054] (2) Design an automatic collaborative method for CIM calculation and CPU calculation

[0055] like Figure 2As shown, the runtime environment of the storage location-aware programming system design method for RISC-V in-memory computing chips proposed in this invention divides computation into two categories: CPU internal computation and CIM external computation. CPU internal computation includes scalar operations, control flow processing, etc., which are directly executed by the RISC-V-based CPU core, and their execution time is short and predictable. CIM external computation includes CIM matrix multiplication, DMA data transfer, etc. These computations are initiated by the CPU through extended instructions, but the actual execution occurs in the CIM array or DMA engine, and their execution time is relatively long and can be performed asynchronously with the CPU. The core design idea of ​​this invention is that from the CPU's perspective, CIM computation is essentially a kind of "external computation," that is, the CPU issues instructions to start CIM computation, but the actual CIM computation process is opaque to the CPU. CIM matrix operations, SPAD data transfer, etc., can be regarded as opaque external computation units. By default, these "external computations" do not block computation on the CPU unless explicit data or control dependencies are generated. During CIM computation, the CPU can continue to execute other dependency-free computations without waiting.

[0056] Based on the above ideas, this invention designs a three-step collaborative execution process of "dispatching tasks - backfilling results - removing dependencies," which interprets and evaluates functions marked with specific decorators in the Python DSL through the runtime environment.

[0057] Step 1: Task Dispatch. When the runtime environment executes a CIM statement, the statement does not block the CPU to wait for the CIM calculation to complete. Instead, it encapsulates the statement as a CIM task and submits it to the CIM task execution queue. Simultaneously, a task placeholder is set for the result variable of this statement in the program's execution state. The CPU then continues to check and execute subsequent statements that do not depend on this task marker.

[0058] Step 2: Backfilling Results. After the CIM hardware completes a task, it notifies the runtime environment via hardware interrupt or polling mechanism. The runtime environment replaces the corresponding task placeholder with the actual result tensor generated by CIM calculation, updates the storage location information of the result tensor, and marks the task as "completed". This step is called "backfilling" because it fills the previously placed placeholder with the actual returned result data.

[0059] Step 3: Dependency Removal. After backfilling the results, the runtime environment rescans the queue of statements to be executed, checking if any statements that were previously blocked due to their dependence on the task's result have become executable. This is done by decrementing the "unresolved dependency" counter for statements that depend on the task; if it reaches zero, it is marked as "stepable". If there are "stepable" statements, they are added back to the executable set and executed in the next interpretation and evaluation step.

[0060] As can be seen from the above design, the runtime environment essentially interprets and evaluates functions marked with the `@cim_kernel` decorator in the Python DSL. It needs to maintain a statement execution status table, recording the status of each statement in the program: "Not Ready," "Steppable," and "Completed." In each interpretation and evaluation step, the runtime environment executes the following decision process: First, it iterates through the incomplete statements, checking whether each statement has been parsed into actual data; if so, it marks it as "Steppable." Then, in the same evaluation step, it executes all steppable statements. For CIM calculation type statements, it performs a "task dispatch" action, sending the statement to the CIM hardware and setting the result as a placeholder; for CPU calculation type statements, it directly interprets and evaluates them. After completing one round of execution, it checks if there are any incomplete statements; if so, it proceeds to the next round of interpretation and evaluation.

[0061] The advantage of this evaluation method is that developers do not need to explicitly control parallelism. The runtime environment automatically discovers parallel opportunities based on data dependencies, and there is no need to perform performance modeling on CIM calculations, facilitating the adaptation of the programming system to different actual hardware. The runtime environment maintains an "unresolved dependency" counter for each statement on the data flow graph converted from the Python DSL program. When a predecessor node completes and backfills its result, the counter for that statement is decremented by one. When it reaches zero, it indicates that it has entered a "stepable" state. When dispatching CIM tasks, the runtime environment interacts with the underlying hardware through an abstract interface, performing operations such as instruction control and data transfer. In systems that support hardware interrupts, an interrupt is triggered after the CIM array completes the calculation. The runtime environment's interrupt handling records the completion status and updates the placeholder markers. In systems that do not support interrupts, the runtime environment can poll the CIM calculation status during each evaluation step to check the task completion status. The runtime environment can also explicitly wait for the CIM task to complete when all "not ready" statements have data dependencies through synchronization primitives with the CIM chip. These mechanisms are transparent to the upper-level evaluation logic, reducing the burden on developers to manually manage parallel tasks.

[0062] (3) Design an automatic resource scheduling method between computational tasks

[0063] The above three-step collaborative execution process conceptually assumes that all CIM computation task dispatching and result backfilling are asynchronous and have zero overhead. However, in actual implementation, true asynchronous elimination requires the CPU to create additional execution contexts (such as threads or coroutines) to monitor the completion status of CIM tasks and handle subsequent logic. The creation and management of these contexts themselves have non-negligible overhead. If the CIM computation time is very short (e.g., small-scale vector addition), the overhead of creating asynchronous contexts may exceed the time consumption of the CIM computation itself, resulting in a negative benefit from asynchronous dispatch. In this case, synchronous computation would be a better strategy. Therefore, this invention further refines the above three-step collaborative execution process. Ideally, asynchronous parallel execution is used for time-consuming CIM computations, and synchronous execution is used for time-consuming CIM computations, but developers do not need to manually control this granularity.

[0064] This invention proposes a heartbeat integral mechanism, providing a runtime scheduling method that can adaptively make reasonable choices between synchronous and asynchronous operations without predicting the CIM calculation time. When interpreting and evaluating functions marked with the @cim_kernel decorator in the Python DSL, the runtime environment automatically marks all CIM calculations as "potentially parallel," performing them serially (i.e., synchronously) by default. After the CPU issues a CIM instruction, it waits for the result in place. However, the runtime environment maintains three different versions of subsequent execution paths (implemented as jump target addresses) in the call stack frame: the first is the serial version, which is the default execution path; the second is the parallel version, which is the execution path of the current CPU thread after the result placeholder of the CIM task when the runtime scheduler decides to make the CIM calculation asynchronous; and the third is the CIM version, which represents the execution path that waits for the asynchronous CIM calculation to complete and backfill the result.

[0065] The runtime environment maintains a global integral manager, implementing a "heartbeat" through timer interrupts in the RISC-V core. The interrupt interval N and the integral C gained per heartbeat are configurable global parameters. Every N microseconds, the runtime environment distributes C integrals to each active CPU thread via an interrupt handler. These integrals are recorded in each thread's local state. When a thread's accumulated integrals reach a certain threshold (i.e., it has been executing serially for a considerable time, indicating a potential for parallel processing), the runtime environment selects the thread with the longest existing integral (i.e., the oldest potential parallel call), consumes the threshold number of integrals, creates a CIM version execution path to wait for the CIM calculation to complete and the result to be filled back, and the current thread marks the result as a placeholder and switches the code execution path to the concurrent version, continuing to execute dependent statements concurrently. When the asynchronously executed CIM task is completed, the runtime environment notifies the original thread via a synchronization primitive or interrupt mechanism, and the CIM version execution path writes back the result and clears the placeholder, assisting the runtime environment in de-depending on other threads.

[0066] The parameters N and C of the heartbeat integration mechanism can be optimized based on the specific CIM chip hardware characteristics and application load. They can be specified by developers or adjusted based on actual performance data. Generally, a larger heartbeat interval parameter N may reduce parallelism and hardware utilization; a smaller parameter N results in more frequent integration checks and decision calls, potentially increasing scheduling overhead. A larger integration parameter C for each heartbeat distribution increases the probability of parallelization, while a smaller parameter C increases the probability of serialization. By fine-tuning parameters N and C, the hardware's parallel potential can be fully utilized, and the overall scheduling mechanism is transparent to developers, reducing the programming burden.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A design method for a memory location-aware programming system for RISC-V in-memory computing chips, characterized in that, The steps of the method include: (1) Design a storage location-aware type system and Python DSL; The physical storage location of data in the RISC-V in-memory computing chip is elevated to a first-class attribute in the programming language type system. Developers write CIM calculation core functions in the Python DSL front-end using decorators and type annotations provided by Python DSL. The type annotations include storage location type annotations, which are used to explicitly declare the physical storage location of data in the multi-level storage hierarchy of the CIM chip. Based on the three-level storage hierarchy involved in the CIM chip, the values ​​of the storage location type annotations include PSRAM, SPAD, or CIM. In the storage location-aware type system, constraint rules for the storage location are defined for each type of CIM calculation. The program statements of the CIM calculation core function are traversed to extract the storage locations of operands related to CIM calculation. These locations are then compared with predefined constraint rules to perform a static storage location validity check. If the current storage location of the operand satisfies the constraint rules, the type system inserts transport primitives as needed and constructs a data flow graph. If the constraint rules are not satisfied, the type system automatically transports and inserts data according to preset transport rules and constructs the data flow graph again. Finally, the data flow graph is passed to the runtime environment. Each node in the data flow graph represents a statement, and each directed edge represents a data dependency. (2) Design an automatic collaborative method for CIM calculation and CPU calculation; Design a runtime environment adapted to the aforementioned Python DSL. This runtime environment divides computation into CPU computation and CIM computation. CPU computation is directly executed by a RISC-V based CPU core, while CIM computation is initiated by the CPU through extended instructions, but the actual execution occurs in the CIM array or DMA engine. The runtime environment then interprets and evaluates functions marked with specific decorators in the Python DSL, including the following three steps: Step 1: Dispatch the task; When the runtime environment executes a CIM statement, it encapsulates the statement into a CIM task and submits it to the CIM task execution queue. At the same time, a task placeholder is set for the result variable of the statement in the program's execution state. The CPU then continues to check and execute subsequent statements that do not depend on the task marker. Step 2: Backfilling results; After the CIM hardware completes a task, it notifies the runtime environment through hardware interrupt or polling mechanism. The runtime environment replaces the corresponding task placeholder with the actual result tensor generated by CIM calculation, updates the storage location information of the result tensor, and marks the task as "completed". Step 3: De-dependency; After backfilling the results, the runtime environment rescans the queue of statements to be executed to check if any statements that were previously blocked due to their dependence on the results of this task have become executable. The check is performed by decrementing the "unresolved dependency" counter of the statements that depend on this task. If the counter reaches zero, the statements are marked as "steppable". If there are "steppable" statements, they are added back to the executable set and executed in the next interpretation and evaluation step. When the runtime environment interprets and evaluates functions marked with specific decorators in the Python DSL, it simultaneously maintains a statement execution status table, recording the status of each statement in the program: "Not Ready", "Step-Ahead", and "Completed". It also maintains a "Unresolved Dependencies" counter for each statement based on the data flow graph. When one of its predecessor nodes completes and backfills its result, the counter for that statement is decremented by one. When it reaches zero, it indicates that it has entered the "Step-Ahead" state. The runtime environment executes the following decision process in each interpretation and evaluation step: First, iterate through the incomplete statements and check if each statement has been parsed into actual data. If so, mark it as "steppable". Then, in the same evaluation step, execute all "steppable" statements. For CIM calculation type statements, perform the "dispatch task" action, that is, send the statement to the CIM hardware and set the result as a placeholder. For CPU calculation type statements, directly interpret and evaluate them. After completing one round of execution, determine if there are any incomplete statements. If so, proceed to the next round of interpretation and evaluation. (3) Design an automatic resource scheduling method between computational tasks; This automatic resource scheduling method, in conjunction with the automatic coordination method between CIM computing and CPU computing, is used to optimize the parallel granularity between computing tasks; When interpreting and evaluating functions marked with specific decorators in the Python DSL, the runtime environment automatically marks all CIM computations as "potentially parallel," performing them serially and synchronously by default. After the CPU issues a CIM instruction, it waits for the result in place. At the same time, the runtime environment maintains three different versions of the subsequent execution path in the call stack frame: the first is the serial version, which is the default execution path; the second is the parallel version, which is the execution path of the current CPU thread after the result placeholder of the CIM task when the runtime scheduler decides to make the CIM computation asynchronous; and the third is the CIM version, which represents the execution path that waits for the asynchronous CIM computation to complete and backfill the result. The runtime environment maintains a global integral manager, implementing a "heartbeat" through a timer interrupt in the RISC-V core. The interrupt interval N and the integral C obtained with each heartbeat are configured. Every N microseconds, the runtime environment distributes C integrals to each active CPU thread via an interrupt handler. These integrals are recorded in each thread's local state. When a thread's accumulated integrals reach a certain threshold, the runtime environment selects the thread with the longest accumulated integrals, consumes the threshold number of integrals, and creates a CIM version execution path to wait for the CIM calculation to complete and backfill the result. The current thread marks the result as a placeholder and switches the code execution path to the concurrent version, continuing to concurrently execute dependent statements. When the asynchronously executed CIM task is completed, the runtime environment notifies the original thread via a synchronization primitive or interrupt mechanism. The CIM version execution path then writes back the result and clears the placeholder, assisting the runtime environment in de-depending on other threads.

2. The method as described in claim 1, characterized in that, The storage location type label corresponds to three storage levels: global storage PSRAM, on-chip temporary storage SPAD, and CIM computing array. When the value is PSRAM, it means that the data is stored in global storage, which has a large capacity but high access latency. This type carries the base address information and capacity constraints of global storage. When the value is SPAD, it means that the data is located in the on-chip temporary storage area, which has the characteristics of high bandwidth and low latency. The value is of type CIM, indicating that the data has resided in the CIM computing array, thus achieving coupling between storage and computing; The three-level storage space is connected by three data paths: the first is a DMA transfer path between PSRAM and SPAD, used to transfer data between global storage and on-chip temporary storage; the second is a weight loading path from SPAD to CIM array, used to write weight data from SPAD in the on-chip temporary storage area to a specified page in CIM array. The third path is the result output path from the CIM array to the SPAD, which is used to write the result to the SPAD used for output buffering after the CIM array has completed the calculation.

3. The method as described in claim 1, characterized in that, The constraint rules for defining the storage location for each type of CIM calculation include: (1) Matrix multiplication requires the weight matrix to be located in the CIM array, i.e., labeled as CIM type; the activation vector or matrix to be located in the on-chip temporary storage area, i.e. labeled as SPAD type; the calculation result to be output to the on-chip temporary storage area, i.e. labeled as SPAD type; convolution operation requires both input and output to be labeled as SPAD type. (2) Vector operations require all operands and outputs to be located in the on-chip temporary storage area SPAD, i.e., marked as SPAD type; (3) Data transfer itself is also subject to storage location constraints. The source of DMA loading is PSRAM and the destination is SPAD; the storage direction of DMA is opposite; the source of CIM writing is SPAD and the destination is CIM.

4. The method as described in claim 1, characterized in that, The system of this type automatically moves and inserts data according to preset moving rules, specifically including: (1) The operand is located in PSRAM and needs to be used in SPAD; the type system first allocates a buffer of the same size in SPAD for the tensor, then inserts a DMA load instruction to move the data from PSRAM to the allocated SPAD buffer, and finally updates the reference of the operand in the statement to the SPAD buffer. (2) The operand is located in SPAD and needs to be used in the CIM array; the type system allocates enough CIM pages for the tensor in the CIM array, then inserts a CIM write instruction to write the data from SPAD to the CIM page, and finally updates the operand reference; (3) The operands are located in PSRAM and need to be used in the CIM array. Since there is no direct path between PSRAM and CIM, the type system needs to perform two steps of data transfer: first, insert a DMA load instruction to transfer the data from PSRAM to SPAD, and then insert a CIM write instruction to load the data from SPAD to the CIM page. When allocating SPAD buffers and CIM pages, the type system maintains a resource tracking table based on hardware configuration information, records currently occupied SPAD units and CIM pages, and manages them according to the user-configured allocation strategy. When resources are insufficient, the type system performs a lightweight lifecycle analysis and recycles unused SPAD buffers or CIM pages.

5. The method as described in claim 1, characterized in that, The specific decorator for the Python DSL is the @cim_kernel decorator.

Images