Cache management methods, devices, computing chips, equipment and media

By explicitly managing dirty bit status and block status in software, the complexity and high power consumption of hardware-based automatic cache consistency protocols are resolved. This enables flexible data management and efficient utilization of storage resources, making it suitable for on-chip storage environments shared by multiple computing units.

CN123068320APending Publication Date: 2026-11-10SUZHOU YIZHU INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611496047.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-24
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In existing technologies, hardware-automated cache coherency protocols suffer from high complexity, high power consumption, and inability to adapt to flexible partitioning in software-managed caches, especially lacking an effective cache coherency maintenance mechanism when multiple computing units share on-chip storage.

Method used

The cache coherence method, which is explicitly managed by software, maintains the dirty bit status and block status of each data block on the on-chip storage, thereby achieving data synchronization and lifecycle management between on-chip and off-chip storage, including space reclamation, defragmentation, and storage mapping relationship management.

Benefits of technology

It reduces the complexity and power consumption of hardware design, improves the flexibility of data management and the efficiency of storage resource utilization, and enables fine-grained control over the lifecycle and consistency of on-chip storage data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068320A_ABST
    Figure CN123068320A_ABST
Patent Text Reader

Abstract

This application provides a cache management method, apparatus, computing chip, device, and medium. The method maintains the dirty bit state and block state of on-chip data blocks to track the synchronization and lifecycle stages of data between on-chip and off-chip storage in real time. When a data allocation request is received, if a free block exists in on-chip storage that meets the conditions, a mapping relationship is established with the off-chip storage, and the block state is updated to active and the dirty bit state is updated to on-chip expired. If no free block exists, a space reclamation operation is performed, selecting an active block as the target, releasing space, and updating it to an evicted state. If the target block's dirty bit is in an off-chip expired state, it is first written back to off-chip storage to ensure no data loss. This application replaces complex hardware consistency logic with a software-controllable dirty bit and block state management mechanism, improving data management flexibility and enhancing the utilization efficiency of computing chip storage resources.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of processor technology, and in particular to a cache management method, apparatus, computing chip, device, and medium. Background Technology

[0002] In computing architectures employing a tiered storage system, to balance the extreme demands of computing units for high bandwidth and low latency, a two-tiered storage system is typically used: on-chip storage and off-chip storage. For example, high-speed scratchpad memory (SPM) is used as on-chip storage, while large-capacity dynamic random access memory (DRAM) is used as off-chip storage. Maintaining data consistency in SPM, as a software-managed cache, is a key challenge in system design. In related technologies, architectures such as graphics processing units (GPUs) generally employ hardware-automated cache coherency protocols, requiring no software involvement. However, this approach suffers from complex hardware logic, high power consumption, and is unsuitable for the management requirements of software-managed SPM. Furthermore, traditional SPM management schemes in digital signal processors or embedded systems typically only handle single-level allocation and deallocation, lacking a cache coherency maintenance mechanism between SPM and DRAM, and also failing to support flexible partitioning allocation when multiple computing units share an SPM. Summary of the Invention

[0003] This application provides a cache management method, apparatus, computing chip, device, and medium, which aims to avoid the complexity of hardware logic control, improve data management flexibility, and enhance the utilization efficiency of computing chip storage resources.

[0004] In a first aspect, embodiments of this application provide a cache management method applied to a computing chip including on-chip storage, wherein the computing chip is connected to off-chip storage, and the method includes:

[0005] Maintain the dirty bit status and block status of each on-chip data block in the on-chip storage. The dirty bit status is used to indicate data consistency between on-chip storage and off-chip storage, and the block status is used to indicate the lifecycle stage of the on-chip data block.

[0006] Receive data allocation requests for on-chip storage and determine the data space to be allocated based on the data allocation requests;

[0007] If there are no free blocks in the on-chip storage that satisfy the data space to be allocated, a space reclamation operation is performed. The space reclamation operation includes: selecting at least one on-chip data block with an active state as the target on-chip data block, releasing the storage space occupied by the target on-chip data block, and updating the block state of the target on-chip data block to the evicted state. When the dirty bit state of the target on-chip data block is the off-chip expired state, the data of the target on-chip data block is written back to the off-chip storage. The off-chip expired state indicates that the data in the on-chip storage has been modified and has not been synchronized to the off-chip storage.

[0008] If there are free blocks in the on-chip storage that satisfy the data space to be allocated, the free blocks are allocated to the data allocation request, a storage mapping relationship is established between the free blocks and the corresponding addresses in the off-chip storage, the block status of the free blocks is updated to active status, and the dirty bit status is updated to on-chip expired status. The on-chip expired status indicates that the data in the off-chip storage has been modified and needs to be reloaded to the on-chip storage.

[0009] In the cache management method provided in this application embodiment, if there are free blocks in the on-chip storage that satisfy the data space to be allocated, the free blocks are allocated to the data allocation request, including:

[0010] If there are contiguous free blocks in the on-chip storage that satisfy the data space to be allocated, the contiguous free blocks are directly allocated to the data allocation request.

[0011] In the cache management method provided in this application embodiment, if there are free blocks in the on-chip storage that satisfy the data space to be allocated, the free blocks are allocated to the data allocation request, and the method further includes:

[0012] If the total free capacity of the on-chip storage is sufficient to fill the data space to be allocated, and there are no contiguous free blocks that fill the data space to be allocated, perform a defragmentation operation on the on-chip storage to obtain contiguous free blocks that fill the data space to be allocated, and allocate the contiguous free blocks to the data allocation request.

[0013] Defragmentation operations include:

[0014] On-chip data blocks that are in an active state and / or a locked state are designated as data blocks to be processed.

[0015] The data blocks to be processed are compactly arranged in on-chip storage according to their address order, and the storage mapping relationship is updated.

[0016] Organize the continuous free areas formed by compact arrangement into at least one continuous free block.

[0017] In the cache management method provided in this application embodiment, before the data blocks to be organized are compactly arranged in on-chip storage according to address order, the method further includes:

[0018] Write the data of the data block to be cleaned, whose current dirty bit status is expired off-chip, from on-chip storage back to off-chip storage, and update the dirty bit status of the data block to be cleaned to a consistent status. A consistent status means that the data in the on-chip data block is consistent with the data in the off-chip storage.

[0019] In the cache management method provided in this application embodiment, the computing chip further includes multiple computing units, which determine the data space to be allocated based on the data allocation request, including:

[0020] If the access mode of the data allocation request is independent mode, the number of computing units and the size of the data to be allocated indicated by the data allocation request are multiplied to obtain the data space to be allocated.

[0021] If the access mode of the data allocation request is shared mode, the size of the data to be allocated indicated by the data allocation request is aligned according to the preset alignment bytes, and the aligned data size is used as the data space to be allocated.

[0022] In the cache management method provided in this application embodiment, the dirty bit state includes a consistent state, an on-chip expired state, and an off-chip expired state; when the data corresponding to the on-chip data block stored off-chip is modified, the dirty bit state of the on-chip data block is updated to the on-chip expired state.

[0023] In the cache management method provided in this application embodiment, when the data stored on the chip is modified, the dirty bit status of the on-chip data block corresponding to the modified data is updated to the off-chip expired status.

[0024] In the cache management method provided in this application embodiment, when there is an on-chip data block with a dirty bit state of on-chip expiration, data is loaded from off-chip storage to on-chip storage based on the storage mapping relationship, and the dirty bit state of the on-chip data block is updated to a consistent state.

[0025] In the cache management method provided in this application embodiment, when there is an on-chip data block with a dirty bit state of off-chip expiration, the data of the on-chip data block is written back to off-chip storage based on the storage mapping relationship, and the dirty bit state of the on-chip data block is updated to a consistent state.

[0026] The cache management method provided in this application embodiment also includes:

[0027] In response to a lazy release request for an on-chip data block, update the block status of the on-chip data block to the idle state;

[0028] When the associated event corresponding to an on-chip data block in an idle state is completed, the storage space occupied by the on-chip data block in an idle state is released. The associated event represents a data access event, data transfer event, or computing unit execution event related to the on-chip data block.

[0029] The cache management method provided in this application embodiment also includes:

[0030] In response to a lock request for an on-chip data block, the block status of the on-chip data block is updated to a locked state, which indicates that the on-chip data block is locked in on-chip storage;

[0031] In response to an unlock request for an on-chip data block, the block status of an on-chip data block that is in a locked state is restored to an active state.

[0032] The cache management method provided in this application embodiment also includes:

[0033] When it is necessary to access an on-chip data block whose block status is obsolete, the data is loaded from off-chip storage to on-chip storage based on the storage mapping relationship, and the block status of the on-chip data block is restored from obsolete to active. The dirty bit status is updated to consistent. The obsolete on-chip data block represents a free block generated after space reclamation operation.

[0034] In the cache management method provided in this application embodiment, the dirty bit status and block status of on-chip data blocks are maintained and updated by explicitly setting through a predefined software interface.

[0035] Secondly, this application also proposes a cache management device, including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing communication between the processor and the memory. The program is executed by the processor to implement the above-mentioned cache management method.

[0036] Thirdly, this application also proposes a computing chip, including on-chip storage, multiple computing units, and the aforementioned cache management device, wherein the computing chip is connected to off-chip storage.

[0037] Fourthly, this application also proposes an electronic device including the aforementioned computing chip.

[0038] Fifthly, embodiments of this application also propose a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the cache management method described above.

[0039] The cache management method, apparatus, computing chip, device, and medium provided in this application maintain the dirty bit status and block status of on-chip data blocks, thereby enabling real-time tracking of data synchronization and lifecycle stages between on-chip and off-chip storage, providing accurate status basis for subsequent storage allocation and reclamation decisions. It receives data allocation requests for on-chip storage, determines the data space to be allocated based on the data allocation request, and searches for free blocks in on-chip storage that satisfy the data space to be allocated. If no free blocks satisfy the data space to be allocated exist in on-chip storage, a space reclamation operation is performed. The space reclamation operation includes: selecting at least one on-chip data block with an active block status as the target on-chip data block, releasing the storage space occupied by the target on-chip data block, and updating the block status of the target on-chip data block to an obsolete state. When the dirty bit status of the target on-chip data block is an off-chip expired state, the data of the target on-chip data block is written back to off-chip storage, ensuring that modified data is not lost while releasing space, thus guaranteeing data consistency under limited on-chip storage resources. If a free block exists, or after a space reclamation operation, that satisfies the data space to be allocated, the free block is allocated to the data allocation request. A storage mapping relationship is established between the free block and the corresponding address in off-chip storage. The block status of the free block is updated to active, and the dirty bit status is updated to on-chip expired. This explicitly indicates that the data block is currently in a state where data needs to be loaded from off-chip storage, so that when executing a data allocation request, data is loaded from off-chip storage to the free block based on the storage mapping relationship. Therefore, this application introduces a software-controllable dirty bit status and block status management mechanism, changing the cache consistency maintenance logic to software explicit control. This achieves control over the lifecycle and consistency status of on-chip storage data, thereby eliminating the dependence on complex hardware consistency logic, improving data management flexibility, and enhancing the utilization efficiency of computing chip storage resources.

[0040] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0041] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0042] Figure 1 This is a schematic diagram of the computing chip architecture provided in an embodiment of this application;

[0043] Figure 2 This is a flowchart illustrating the cache management method provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the overall process of the cache management method provided in the embodiments of this application;

[0045] Figure 4 This is a schematic diagram illustrating the specific process of space reclamation operation provided in the embodiments of this application;

[0046] Figure 5 This is a flowchart illustrating the defragmentation operation provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of an optional structure of the cache management device provided in the embodiments of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0050] To facilitate understanding of the technical solutions provided in the embodiments of this application, some key terms used in the embodiments of this application will be explained below:

[0051] Compute-in-Memory (CIM) architecture refers to a system architecture that tightly integrates computing and storage units to reduce data transfer latency and power consumption. In this architecture, data processing and storage operations are physically closer together.

[0052] On-chip memory (SPM) refers to memory units integrated inside a computing chip. They typically feature high speed and low latency, but their capacity is relatively limited. Their function is to serve as a fast cache or temporary storage area for the computing unit.

[0053] Off-chip DRAM refers to memory units located outside the computing chip. It typically has a large capacity but relatively high access latency. Its function is to serve as the system's main memory.

[0054] A data allocation request is an instruction issued by a computing unit or software module to the storage management system, requesting the allocation of a specific size of storage space in the on-chip storage to store data.

[0055] Block state refers to the state of an on-chip data block at different stages of its lifecycle, such as active, eviction, idle, and locked. These states are used to manage the availability of on-chip data blocks and their lifecycle in on-chip storage.

[0056] The dirty bit status refers to a flag used to indicate the data consistency between on-chip data blocks and corresponding data blocks in off-chip storage. For example, the off-chip expired status indicates that the on-chip data block has been modified but has not yet been written back to off-chip storage; the on-chip expired status indicates that the data in the on-chip data block is inconsistent with the data in off-chip storage, and the latest data needs to be loaded from off-chip storage; the consistent status indicates that the data in the on-chip data block is consistent with the data in off-chip storage.

[0057] In architectures employing a tiered system of on-chip and off-chip storage, a tiered storage system is typically used to balance the high bandwidth and low latency requirements of computing units. As a software-managed cache, SPM's data consistency maintenance is a key challenge in system design. Traditional technologies commonly employ hardware-automated cache consistency protocols. However, this mechanism is transparent to software, preventing developers from implementing fine-grained data flow control based on application characteristics, and also resulting in complex hardware logic and high power consumption.

[0058] Based on this, embodiments of this application propose a cache management method, apparatus, computing chip, device, and medium to avoid the complexity of hardware logic control, while improving data management flexibility and enhancing the utilization efficiency of computing chip storage resources.

[0059] Reference Figure 1 , Figure 1 This is a schematic diagram of the architecture of the computing chip provided in the embodiments of this application. As shown in the figure, the computing chip in the architecture where on-chip storage is explicitly managed by software includes a two-level cache allocator located between the application layer and the storage layer. It is responsible for the software-managed consistent cache allocation of on-chip storage SPM and off-chip storage DRAM in the architecture where on-chip storage is explicitly managed by software. Specifically, the application layer initiates an allocation request through a standardized interface, and through the collaborative decision-making of the core management module and the policy module, the data is efficiently migrated and synchronized between SPM and DRAM.

[0060] First, the application layer represents the computational tasks of the upper layers, such as neural network inference and training, which require the allocation of SPM space to accelerate data access. The tensor operation interface is a predefined software interface in the allocator. The application layer can interact with the allocator through unified APIs such as malloc(), free(), pin(), unpin(), and flushToDram() to achieve explicit software management.

[0061] Secondly, the core management module includes a block manager, a block state machine, and a dirty bit tracker. The block manager maintains a mapping table between off-chip storage addresses and on-chip data blocks, i.e., a storage mapping relationship. This mapping establishes an association between data blocks in on-chip storage and their corresponding data addresses in off-chip storage. Through this mapping, the block manager can determine the location in off-chip storage corresponding to an on-chip data block. The block manager records the starting address pointer, allocation size, status, stream, and event for each on-chip data block. The block manager is the core data structure of the allocator; all allocation, release, and lookup operations are performed through the block manager, which is fundamental to data consistency and space management.

[0062] The block state machine defines four lifecycle states for on-chip data blocks, driving their allocation, release, eviction, residency, and recovery. Specifically, the four block states are: FREE, OUTTED, USING, and PINNED. The FREE state indicates that the on-chip data block is in a lazy-release state, waiting for the associated event to end before space reclamation. At this time, the data in the on-chip data block may still be in on-chip storage, but it has been logically released. The OUTTED state indicates that the data in the on-chip data block has been evicted from on-chip storage, written back to off-chip storage, and the space occupied in on-chip storage has been released and can be reallocated. However, the storage mapping relationship is still retained and can be recovered. The USING state indicates that the on-chip data block is currently being used by a computing unit or is in a valid state and can be evicted. The PINNED state indicates that the on-chip data block is locked and resides in on-chip storage and cannot be evicted.

[0063] The dirty bit tracker uses three dirty bit flags to track the data consistency status between on-chip and off-chip memory. Specifically, the three dirty bit flags are MEM_DIRTY_NONE, MEM_DIRTY_DRAM, and MEM_DIRTY_SPM. The MEM_DIRTY_NONE dirty bit flag indicates a consistent dirty bit status, meaning the data in the off-chip memory and on-chip memory are consistent and no synchronization operation is required. The MEM_DIRTY_DRAM dirty bit flag indicates an on-chip expired dirty bit status, meaning the data in the on-chip data block is inconsistent with the data at the corresponding address in the off-chip memory, and the data in the off-chip memory is the latest version, requiring data to be loaded from the off-chip memory into the on-chip data block to maintain data consistency. The MEM_DIRTY_SPM dirty bit flag indicates an off-chip expired dirty bit status, meaning the data in the on-chip data block has been modified and is inconsistent with the data at the corresponding address in the off-chip memory, requiring the data in the on-chip data block to be written back to the off-chip memory to maintain data consistency.

[0064] In this system, the dirty bit state and block state of on-chip data blocks are not maintained automatically by hardware, but are explicitly maintained and updated through predefined software interfaces. These predefined software interfaces refer to a set of standardized application programming interfaces (APIs) that allow upper-layer software to directly operate and manage underlying hardware or allocator resources by calling these interfaces, such as tensor operation interfaces provided by the application layer. These interfaces can be represented as a set of function calls, allocator calls, or an abstraction layer for read / write operations through memory-mapped registers. Explicitly setting these interfaces for maintenance and updates means that the software actively calls specific interfaces or instructions to directly specify and modify the dirty bit state and block state of on-chip data blocks. Unlike hardware automatic detection or implicit update mechanisms, explicit setting gives the software control over state management. For example, by calling specific API functions, the dirty bit state can be set to an off-chip expired state, or the block state can be updated to a locked state. The software can convey its intent regarding data state through these predefined software interfaces, and the underlying hardware or cache management logic performs corresponding data loading, write-back, eviction, or allocation operations based on these explicitly set states.

[0065] Therefore, by explicitly setting the dirty bit state and block state of on-chip data blocks through a predefined software interface, the problem of developers being unable to perform fine-grained data flow control based on application characteristics due to implicit state maintenance or automatic hardware maintenance in traditional cache management is solved. This allows cache management strategies to be deeply coupled with specific computing tasks, enabling developers to flexibly adjust dirty bit update strategies and block state transition strategies according to application scenarios, achieving more targeted data flow control and storage resource scheduling. For example, after a computing unit modifies an on-chip data block, the software can explicitly set the dirty bit state to an off-chip expired state, ensuring data consistency and avoiding the complexity and potential latency of automatic hardware detection. Simultaneously, by explicitly setting the block state, on-chip storage space can be effectively managed, such as locking critical data blocks to prevent their eviction, thereby effectively improving the allocator's flexibility, programmability, and overall performance. Especially in architectures where on-chip storage is explicitly managed by software, it can better balance the computing unit's needs for high bandwidth and low latency.

[0066] The strategy module is the decision-making center of the allocator, responsible for adjusting the allocation strategy based on the current state. Specifically, the strategy module includes an eviction policy, a step allocator, a defragmenter, and a lock manager. The eviction policy selects active on-chip data blocks for eviction when on-chip storage capacity is insufficient, responding to data allocation requests. The step allocator calculates the space to be allocated based on the access patterns of the data to be allocated. The defragmenter is triggered when total free capacity is sufficient but there are no contiguous free blocks; it merges gaps into contiguous free blocks by compactly arranging active and locked on-chip data blocks. The lock manager updates active on-chip data blocks to a locked state, exempting them from space reclamation operations, or updates locked on-chip data blocks to an active state, allowing them to be evicted under space pressure.

[0067] The dual-level cache allocator also includes a synchronization module, which can be understood as the execution layer of the allocator, responsible for data movement. The synchronization module can execute `flushToSpm` and `flushToDram` operations. `flushToSpm` copies data from off-chip storage to on-chip storage, while `flushToDram` writes data from on-chip storage back to off-chip storage. Specifically, the synchronization module determines whether data transfer is necessary based on the dirty bit status of the on-chip data block, and executes the corresponding `flushToSpm` or `flushToDram` operation. After the data transfer is completed, the dirty bit status of the on-chip data block is updated to a consistent state.

[0068] Reference Figure 2 ,based on Figure 1The system architecture shown illustrates the cache management method provided in the embodiments of this application. The cache management method includes, but is not limited to, steps 210 to 240.

[0069] Step 210: Maintain the dirty bit status and block status of each on-chip data block in the on-chip storage;

[0070] Step 220: Receive a data allocation request for on-chip storage, and determine the data space to be allocated based on the data allocation request;

[0071] Step 230: If there are no free blocks in the on-chip storage that satisfy the data space to be allocated, perform a space reclamation operation. The space reclamation operation includes: selecting at least one on-chip data block with an active state as the target on-chip data block, releasing the storage space occupied by the target on-chip data block, and updating the block state of the target on-chip data block to the evicted state. When the dirty bit state of the target on-chip data block is the off-chip expired state, the data of the target on-chip data block is written back to the off-chip storage.

[0072] Step 240: If there are free blocks in the on-chip storage that satisfy the data space to be allocated, allocate the free blocks to the data allocation request, establish a storage mapping relationship between the free blocks and the corresponding addresses in the off-chip storage, and update the block status of the free blocks to the active state and the dirty bit status to the on-chip expired state.

[0073] First, the block manager maintains the dirty bit status and block status of each on-chip data block in the on-chip storage. This status information can be maintained by setting specific flag bits in the metadata area of ​​each on-chip data block, or by recording the status of each data block in a centralized software table. For example, when an on-chip data block is allocated, the block status is set to active and the dirty bit status is set to on-chip expired; when the on-chip data block is modified, the dirty bit status is updated to off-chip expired.

[0074] When a data allocation request for on-chip storage is received, the data space to be allocated is determined based on the request. The data allocation request can be initiated by the computing unit during task execution, or by the operating system or runtime environment. This request contains information about the size of the required data. The data space to be allocated can be determined based on the size directly specified in the data allocation request; for example, if the request specifies 1KB of space, then the allocated data space is 1KB. Alternatively, the allocated data space can also be aligned according to a preset block size; for example, if the on-chip storage uses 256 bytes as the minimum allocation unit, then any request smaller than 256 bytes will be aligned up to 256 bytes.

[0075] When processing a data allocation request, the system checks if there are any free blocks in the on-chip storage that can satisfy the data space to be allocated. A list of free blocks can be traversed to search for such blocks within the on-chip storage. This list can be a data structure maintained by the block manager, recording information such as the starting address and size of all available free areas in the on-chip storage. If no free blocks satisfy the data space to be allocated are found in the on-chip storage, a space reclamation operation is performed. This operation involves selecting at least one on-chip data block with an active state as the target on-chip data block. The selection strategy can be based on various factors; for example, a Least Recently Used algorithm can be used to select the active data block that has not been accessed for the longest time, or a First-In-First-Out (FIFO) algorithm can be used to select the data block that entered the active state earliest. After releasing the storage space occupied by the target on-chip data block, its block state is updated to eviction. During this process, if the dirty bit state of the target on-chip data block is in an off-chip expired state, the data is written back to off-chip storage. For example, if an active data block is modified on-chip and its dirty bit status is off-chip expired, indicating that the corresponding data in off-chip storage has expired, then the latest data needs to be written back to off-chip storage before reclaiming the data block to ensure data consistency.

[0076] After performing space reclamation and releasing storage space, the system will re-traverse the free block list. The purpose of this traversal is to search again for free blocks in the on-chip memory that meet the size to be allocated. For example, the system can start searching again from the beginning of the free block list, or continue searching from the breakpoint of the previous search.

[0077] If a free block exists in the on-chip storage that satisfies the data space to be allocated, or if a free block exists after a re-traversal that satisfies the data space to be allocated, then the free block is allocated to the data allocation request. After allocating the free block, the system establishes a storage mapping relationship between the free block and the corresponding address in off-chip storage. This mapping relationship can be achieved by maintaining an address translation table, which records the correspondence between the physical address in on-chip storage and the logical address in off-chip storage. For example, when an on-chip data block is allocated, its physical address in on-chip storage is associated with a logical address in off-chip storage. At the same time, the block status of the free block is updated to active, indicating that the data block is currently in use; its dirty bit status is updated to on-chip expired, indicating that the data in the data block needs to be loaded from off-chip storage, or that the data in off-chip storage is the latest version.

[0078] The following example will provide a more detailed explanation of the above technical solution:

[0079] Reference Figure 3 , Figure 3This is a schematic diagram of the overall process of the cache management method provided in the embodiments of this application.

[0080] Suppose a computing chip, which can be an in-memory computing architecture, but is not limited to this. This computing chip has limited on-chip storage capacity, while off-chip storage capacity is larger. The allocator needs to handle a computing task involving iterative computation on a large dataset.

[0081] First, the allocator maintains the dirty bit status and block status of each on-chip data block in the on-chip storage. For example, the metadata of each data block contains a dirty bit field (0 for consistency, 1 for on-chip expiration, 2 for off-chip expiration) and a block status field (0 for free, 1 for active, 2 for evicted, 3 for locked).

[0082] Now, computing unit A initiates a data allocation request, requesting 1MB of space in on-chip storage to load a portion of a dataset. Upon receiving this request, the allocator first determines if the storage type is on-chip storage (SPM). If so, it calculates the data space to be allocated based on the requested data size and access mode. Then, the allocator iterates through the free block list, searching for a contiguous free region that satisfies the requirement for the allocated data space.

[0083] In scenario one, if a free block meets the conditions, the allocator will allocate it to computation unit A. Subsequently, the allocator will establish a storage mapping relationship between this free block and the corresponding address of the dataset in off-chip storage. For example, by updating a global mapping table, the physical address of this free block is associated with the logical address range of the dataset in off-chip storage. Simultaneously, the block status of this free block is updated to active, indicating that it is now occupied by computation unit A and ready for use; and the dirty bit status is updated to on-chip expired, indicating that computation unit A needs to load data from off-chip storage into these blocks. Then, the starting address of this free block in on-chip storage is written to its own starting address field and registered in the block manager. Computation unit A can then load the required data from off-chip storage and begin computation according to the mapping relationship.

[0084] Scenario 2: If there is currently no free area in the on-chip storage to satisfy the data space to be allocated, but some on-chip data blocks are occupied by other computing tasks, such as computing unit B, the allocator triggers an eviction policy and performs space reclamation. The allocator will select a number of active on-chip data blocks corresponding to the data space to be allocated, based on a preset eviction policy (e.g., selecting the least recently used active on-chip data blocks). For example, if the data space to be allocated aligns with the data space occupied by four on-chip data blocks previously used by computing unit B, then the allocator can select these four on-chip data blocks.

[0085] For these target on-chip data blocks, the allocator checks their dirty bit status. If the dirty bit status of a target on-chip data block is "off-chip expired," it means that the on-chip data block has been modified on-chip, and the latest data has not yet been written back to off-chip storage. The allocator will first write the data of the on-chip data block back to the corresponding address in off-chip storage to ensure that data is not lost and to maintain data consistency in off-chip storage. After the write-back is complete, the dirty bit status of the on-chip data block can be updated to a consistent state. If the dirty bit status of a target on-chip data block is either "consistent" or "on-chip expired," it means that the on-chip data is consistent with the off-chip data, or the off-chip data is the latest version, and no write-back is required. After completing the necessary write-back operations, the storage space occupied by these target on-chip data blocks is released, and their block status is updated to "erased."

[0086] After the space reclamation operation, the allocator checks again for available free blocks that meet the criteria. If there are now enough free blocks in the on-chip storage, the allocator selects four of these newly freed blocks and allocates them to compute unit A. Subsequent steps are the same as in scenario one: establishing a storage mapping relationship, updating the block status to active, and updating the dirty bit status to on-chip expired. Computation unit A can then load data and perform calculations. If no free blocks are available—for example, if the total free capacity is sufficient but the free blocks are not contiguous—the allocator triggers a defragmentation operation. If, after defragmentation, available free blocks meet the criteria, the allocation is successful, and subsequent steps are the same as in scenario one. If the total free capacity is insufficient, the allocation fails, and an allocation exception notification is returned.

[0087] In the example above, when computing unit A requests space allocation, the allocator can make intelligent decisions based on the block status (e.g., active, eviction) and dirty bit status (e.g., off-chip expired, on-chip expired). For example, when space is insufficient, the allocator can accurately identify which active data blocks can be reclaimed, and before reclamation, it can determine whether data needs to be written back to off-chip storage based on the dirty bit status. This fine-grained management based on software state allows developers to flexibly adjust the lifecycle and consistency maintenance strategies of data blocks according to actual application needs, thereby achieving customized control over the on-chip storage data flow.

[0088] In traditional computing architectures employing tiered storage systems, cache consistency typically relies on complex hardware protocols for automatic maintenance, as described in the background section. While this hardware mechanism guarantees consistency, its transparency to software prevents developers from implementing fine-grained data flow control based on application characteristics, and the hardware logic is complex and consumes significant power. This application effectively addresses these issues by introducing software-managed dirty bit states and block states. Compared to hardware-automatically maintained consistency protocols, this application's solution replaces consistency maintenance logic with a more flexible software layer, reducing hardware design complexity and power consumption, and improving the allocator's programmability and adaptability. For example, during space reclamation operations, when the dirty bit state of a target on-chip data block is in an off-chip expired state, a data write-back is performed, avoiding unnecessary write-back operations and improving storage access efficiency. This conditional write-back mechanism is a direct advantage of explicit software management, which is difficult to achieve with hardware-transparent mechanisms.

[0089] Furthermore, by establishing a storage mapping relationship between on-chip storage and off-chip storage, this application ensures the correct addressing and loading of data in a hierarchical storage system. When an on-chip data block is allocated and marked as expired, the computing unit can accurately load the required data from off-chip storage according to this storage mapping relationship, ensuring data consistency between on-chip and off-chip storage.

[0090] In summary, the cache management method of this application achieves fine-grained and controllable management of the lifecycle and consistency of on-chip storage data in computing chips by explicitly maintaining the dirty bit state and block state in software. It overcomes the limitations of traditional hardware consistency schemes, reduces allocator complexity and power consumption, and effectively improves the utilization efficiency of storage resources and data consistency.

[0091] Specifically, when the allocator receives a data allocation request and there are no free blocks in the current on-chip storage that satisfy the data space to be allocated, the eviction policy is triggered, and space reclamation is performed. (See reference...) Figure 4 , Figure 4This is a schematic diagram illustrating the specific process of space reclamation operation provided in this application embodiment. First, based on a preset eviction policy, one or more target on-chip data blocks are selected from all currently active (USING) on-chip data blocks. The number of target on-chip data blocks is determined based on the data space to be allocated. For each selected target on-chip data block, the allocator checks its current dirty bit status, specifically determining whether the dirty bit field of the target on-chip data block is MEM_DIRTY_SPM, i.e., whether it is in an off-chip expired state. If the dirty bit field of the target on-chip data block is MEM_DIRTY_SPM, it indicates that the data block has been modified in the on-chip storage SPM, and the latest data has not yet been written back to the off-chip storage DRAM. At this time, the allocator first performs a flushToDram operation, writing the data of the target on-chip data block from the on-chip storage SPM back to the corresponding address in the off-chip storage DRAM to ensure that data is not lost and updating the dirty bit status to a consistent state. If the dirty bit field of the target on-chip data block is not MEM_DIRTY_SPM, it indicates that the dirty bit status of the data block is either MEM_DIRTY_NONE (consistent) or MEM_DIRTY_DRAM (on-chip expired), meaning the data in the SPM is consistent with the data in the DRAM, or the data in the DRAM is the latest version. In this case, no write-back operation is required, and the next step can be performed directly. Regardless of whether a write-back operation has been performed, the allocator will release the SPM physical space occupied by the target on-chip data block and update the block status to OUTTED, indicating that the data block has been evicted from the SPM. The SPM space occupied by the evicted data block is successfully reclaimed, becoming available free space, which the allocator will return for subsequent allocation requests.

[0092] It is important to note that when selecting target on-chip data blocks, it is necessary to ensure that the free storage areas in on-chip memory used to satisfy data allocation requests are physically contiguous. This contiguousness is crucial for computational tasks that require accessing large amounts of data at once or employing specific data structures (such as arrays or matrices), as contiguous storage space improves data access efficiency and reduces memory management overhead. Therefore, when managing on-chip memory, the allocator manages and allocates contiguous free regions, i.e., contiguous free blocks, as the basic allocation unit. A contiguous free block refers to a contiguous free address region in on-chip memory where no on-chip data blocks are in an active or locked state. When the allocator receives a data allocation request for on-chip memory, it first determines the required data space to be allocated based on the request. Then, the allocator checks whether there are free blocks in on-chip memory that not only have the capacity to satisfy the data space to be allocated but are also physically contiguous. When a contiguous free block that meets the contiguousness requirement is found, the allocator directly allocates that contiguous free block to the data allocation request. During the allocation process, a storage mapping relationship is simultaneously established between the free block on the chip and the corresponding address in the off-chip storage to ensure subsequent data synchronization and consistency. At the same time, the block status of the free block is updated to active, indicating that it has been put into use; the dirty bit status is also updated to on-chip expired, meaning that the block is ready to receive new data and requires a subsequent flushToSpm operation to establish consistency.

[0093] In one embodiment, in practical applications, with the frequent occurrence of data allocation requests, on-chip storage space is prone to fragmentation. That is, although the total free capacity meets the requirements of the data space to be allocated, the lack of sufficiently large contiguous free blocks makes it impossible to directly satisfy a specific data allocation request. Therefore, if the total free capacity of the on-chip storage meets the requirements of the data space to be allocated, and there are no contiguous free blocks that meet the requirements of the data space to be allocated, a defragmentation operation is performed on the on-chip storage. The defragmentation operation includes: taking on-chip data blocks in the active state and / or locked state as data blocks to be defragmented; compactly arranging the data blocks to be defragmented in the on-chip storage according to address order and updating the storage mapping relationship; and defragmenting the contiguous free areas formed after compact arrangement into at least one contiguous free block. Here, the total free capacity can refer to the total data size occupied by on-chip data blocks in the obsolete state, the free state, and the active state.

[0094] Reference Figure 5 , Figure 5This is a flowchart illustrating the defragmentation operation provided in this application embodiment. When the allocator receives a data allocation request, it first assesses the free status of the on-chip storage. If the total free capacity of the on-chip storage is sufficient to meet the demand for the data space to be allocated, but a check reveals that no single contiguous free block can meet the demand, then a defragmentation operation is triggered. The defragmentation operation aims to improve the utilization and allocation efficiency of storage space by reorganizing the allocated data blocks in the on-chip storage, merging scattered free spaces into larger contiguous free areas.

[0095] When performing a defragmentation operation, the allocator performs the following steps:

[0096] The first step is to identify and sort the data blocks to be processed. The allocator identifies on-chip data blocks whose current block status is active (USING) and / or locked (PINNED), and marks them as data blocks to be processed. An active status indicates that the data block is being accessed or used by a computing unit, while a locked status indicates that the data block is explicitly pinned in on-chip storage to prevent it from being evicted. These data blocks to be processed are then sorted according to their address order in on-chip storage.

[0097] The second step involves compactly arranging the data blocks to be processed one by one. The allocator arranges these data blocks in the sorted order, compactly placing them into the starting address region of the on-chip storage, ensuring that the on-chip data blocks are stored contiguously in physical addresses without gaps. Before moving each data block, it's necessary to determine if the block meets a specific condition: whether its dirty bit status is off-chip expired and whether it needs to be moved. If the dirty bit status is off-chip expired and movement is required, it means the data block has been modified in SPM, and the latest data has not yet been written back to the off-chip DRAM. In this case, a flushToDram operation is executed to write the data block's data back to DRAM before moving it within SPM, ensuring that no modifications are lost when moving dirty data. If the dirty bit status is not off-chip expired and movement is required, it means the data block's dirty bit status is consistent or on-chip expired, and no write-back is needed; the data can be moved directly within SPM.

[0098] The third step is to update the storage mapping relationships of these data blocks. After the data blocks are moved, the allocator synchronously updates the block mapping table, i.e., the SPM physical address of each data block in the storage mapping relationship. The storage mapping relationship records the association between on-chip data blocks and corresponding addresses in off-chip storage, ensuring that the allocator can access the data through the correct address, maintaining data consistency and accessibility.

[0099] The fourth step is to merge the gaps and reset the free block list. Through the compact arrangement described in the second step, all the free space originally scattered between data blocks is merged into one or more large, contiguous free regions. The allocator merges these contiguous regions into contiguous free blocks and resets the free block list so that it can be recognized and used by subsequent allocation requests.

[0100] The fifth step involves attempting to allocate space from the consolidated free blocks. The allocator attempts to allocate the required data space from the contiguous free blocks formed after consolidation. If the allocation is successful, the allocation result is returned; if it still fails, for example, if the total free capacity is insufficient, the allocation is declared a failure.

[0101] The following example illustrates this. Assume the total on-chip storage capacity is 100 units. Currently, there are three active on-chip data blocks A, B, and C, occupying address ranges [0,9], [20,29], and [40,49] respectively. At this time, the total free capacity of the on-chip storage is 70 units, but the largest contiguous free blocks are only [10,19], [30,39], and [50,99]. If a data allocation request requires 60 units of contiguous storage space, although the total free capacity of 70 units meets the demand, no single contiguous free block can satisfy the 60-unit requirement. In this situation, the allocator will trigger a defragmentation operation. First, since on-chip data blocks A, B, and C are all active, they are selected as data blocks to be defragmented. The allocator then arranges these data blocks in compact order according to their addresses. For example, on-chip data block A is moved to address [0,9], on-chip data block B is moved to address [10,19], and on-chip data block C is moved to address [20,29]. During the move, if the dirty bit status of on-chip data blocks A, B, or C is in an off-chip expired state, their data will be written back to off-chip storage before the move to ensure data consistency. If the dirty bit status is in a consistent state or an on-chip expired state, the move will occur directly within the on-chip storage. After the data blocks are moved, the allocator updates the storage mapping to ensure that the new addresses of on-chip data blocks A, B, and C are correctly associated with their corresponding addresses in off-chip storage. For example, if on-chip data block B was originally mapped to address X in off-chip storage, and its on-chip storage address is now [10, 19], the on-chip address corresponding to off-chip address X in the mapping will be updated to [10, 19]. After compact arrangement, on-chip data blocks A, B, and C occupy addresses [0, 29], while addresses [30, 99] form a contiguous free region of 70 units. This contiguous free region is then consolidated into a contiguous free block, thus fulfilling the previously requested 60 units of contiguous storage space.

[0102] Therefore, by introducing a defragmentation mechanism, the problem of being unable to allocate contiguous storage blocks due to storage space fragmentation in computing chips with on-chip storage explicitly managed by software is effectively solved. When the total free capacity is greater than or equal to the requested size, but the largest contiguous free block is less than the requested size, the allocator initiates a defragmentation operation. It identifies active or locked data blocks as blocks to be defragmented, arranges them compactly at the beginning of the on-chip storage according to their address order, and updates the storage mapping relationship. This merges the gaps originally scattered between multiple data blocks into one or more contiguous free regions, thereby optimizing the on-chip storage layout and ensuring the correctness of data access by updating the storage mapping relationship. By defragmenting fragmented free space into contiguous blocks, the allocator can respond more flexibly to subsequent data allocation requests, thereby improving the utilization efficiency of on-chip storage space, avoiding allocation failures caused by space fragmentation, and enhancing the robustness of cache management.

[0103] It is worth noting that during the defragmentation operation described above, when writing data from data blocks whose current dirty bit status is outdated off-chip to off-chip storage, the dirty bit status of the data blocks to be defragmented needs to be updated to a consistent state before data movement is performed. This ensures that all modifications are persisted before the data block is moved or its storage space is reallocated. When the data is successfully written back, the dirty bit status of the data block will be updated to a consistent state, indicating that the data between on-chip and off-chip storage has been synchronized. This ensures the data integrity and reliability of the dual-level storage allocator, thereby improving the security of computing chip storage management.

[0104] In computing chips with an architecture where on-chip storage is explicitly managed by software, the Storage Processing Unit (SPM) is shared by multiple computing units. However, different computing tasks have different access patterns to the SPM. Therefore, if the access pattern of the data allocation request is independent, the number of computing units and the size of the data to be allocated indicated by the data allocation request are multiplied to obtain the data space to be allocated. If the access pattern of the data allocation request is shared, the size of the data to be allocated indicated by the data allocation request is aligned according to a preset alignment byte, and the aligned data size is used as the data space to be allocated.

[0105] The computing chip contains multiple computing units to perform computational tasks. These computing units can be multiple CPU cores, DSP cores, or specially designed AI accelerator units, such as matrix multiplication units or vector processing units, which can work in parallel to improve overall computational throughput.

[0106] The data allocation request uses an independent access mode, meaning each computing unit requires its own independent, non-overlapping memory region in on-chip storage to store data for parallel processing of different data blocks. For example, when processing different data blocks in parallel, the data processed by each computing unit does not affect others. Specifically, the size of the data to be allocated required by each computing unit indicated in the data allocation request is rounded up according to a preset alignment byte, such as 256 bytes, to obtain the partition size. Then, the partition size is multiplied by the number of computing units to obtain the data space to be allocated. This aims to ensure that sufficient total storage space is reserved for all participating computing units to meet their respective independent data needs. The allocator can maintain a list of active computing units and, upon receiving an independent mode allocation request, queries this list to obtain the number of computing units, which is then multiplied by the requested data size.

[0107] The data allocation request uses a shared access mode, meaning multiple computing units will access the same data region in on-chip memory. In this mode, data is shared among computing units, so only one copy needs to be stored. For example, multiple computing units may need to access the same set of model weights or shared input data. Specifically, the size of the data to be allocated indicated by the data allocation request is rounded up by a preset alignment byte, and the aligned data size is used as the allocated data space, ensuring that the starting address of the allocated memory region is an integer multiple of the preset alignment byte. The preset alignment byte can be configured according to the allocator hardware characteristics or data type; the alignment operation is typically achieved by rounding the original size up to the nearest alignment multiple.

[0108] Alignment operations ensure that the starting address of data in on-chip storage meets the conditions for hardware access, thereby improving data access performance. After determining the data space to be allocated, the allocator will continue to execute subsequent steps in the basic cache management method, such as checking whether there are free blocks in on-chip storage that meet the conditions. If not, space reclamation operations will be performed, and finally the determined data space will be allocated to the request, and the corresponding block status and dirty bit status will be updated. This makes the allocation of on-chip storage more intelligent and can be flexibly adjusted according to the actual needs of computing tasks, thereby achieving more efficient and less conflict-free storage resource management in computing chips with multiple computing units.

[0109] It's important to note that the dirty bit status needs to accurately reflect the data consistency between on-chip and off-chip storage. Therefore, when data in off-chip storage is modified externally, the corresponding data block in on-chip storage becomes stale. If the dirty bit status of the on-chip data block is not correctly updated to the on-chip stale state, the allocator will mistakenly consider the SPM data valid, thus reading stale data and causing calculation errors. Therefore, when the data corresponding to an on-chip data block in off-chip storage is modified, the dirty bit status of the on-chip data block is updated to the on-chip stale state to ensure that the dirty bit status accurately reflects the location of the latest version of the data.

[0110] Changes in the original data corresponding to a data block in on-chip memory and its corresponding data in off-chip memory can be caused by various external factors. For example, the off-chip DRAM may be directly written to by other computing units, DMA controllers, or external I / O devices. One approach is to monitor whether the original data in off-chip memory has changed through hardware monitoring, such as setting up listening logic in the off-chip memory controller to trigger an interrupt or signal when a write operation to a specific address range is detected. Alternatively, software notification can be used to determine whether the data has changed; for example, when an external module completes a write operation to off-chip memory, it notifies the block manager via message passing or a shared status register.

[0111] When it is determined that the data corresponding to the on-chip data block in off-chip storage has been modified, the block manager can query the storage mapping relationship, find the corresponding on-chip data block, and directly modify the dirty bit field of the data block; or, through software instructions, the operation allocator or runtime library can call specific API functions to update the dirty bit status of the on-chip data block after receiving a hardware event or external notification.

[0112] Therefore, when the original data corresponding to an on-chip data block in off-chip storage is modified, the allocator can promptly detect and take action. Specifically, in a computing chip, data consistency between on-chip and off-chip storage is managed by maintaining the dirty bit state of on-chip data blocks. When data in off-chip storage is modified by other entities such as other computing units, DMA controllers, or external devices, the corresponding on-chip data block in on-chip storage is no longer the latest version, and the data is in an outdated state. To prevent subsequent computing units from accessing outdated data, the dirty bit state of the affected on-chip data block is updated from its current state to an on-chip expired state upon detecting the modification event in off-chip storage. The on-chip expired state explicitly indicates that the data in the on-chip data block is no longer valid, and the latest data version resides in off-chip storage. It should be noted that any access request to this on-chip data block will first check its dirty bit state. If it is found to be in an on-chip expired state, the allocator will trigger the operation of loading the latest data from off-chip storage, thereby ensuring that the computing unit always obtains correct and up-to-date data.

[0113] Furthermore, when on-chip data is modified, the dirty bit status of the on-chip data block corresponding to the modified data is updated to an off-chip expired state. Specifically, modification of on-chip data refers to a write operation performed by the computing unit on the data in the on-chip storage, causing a change in its content. This can be done through direct memory access or through instructions executed by the computing unit. When any data in a block is modified, the entire block is considered the on-chip data block corresponding to the modified data. When the dirty bit status of an on-chip data block is set to an off-chip expired state, it indicates that the data in that on-chip data block has been modified, and its content is inconsistent with the data at the corresponding address in off-chip storage; that is, the data in on-chip storage is the latest version, while the data in off-chip storage is an older version. The dirty bit status update operation can be explicitly triggered by software through specific interfaces or instructions.

[0114] Understandably, in computing chips, when a computing unit modifies data in on-chip memory via kernel write operations, the hardware does not automatically update the dirty bit status because SPM is explicitly managed by software. The runtime allocator explicitly sets the dirty bit status of the corresponding on-chip data block to an off-chip expired state through a software interface. This explicit marking of the dirty bit status of the on-chip data block clearly indicates that the data in the on-chip data block is dirty, meaning it is inconsistent with the data in off-chip memory, and that the data in the on-chip data block is the newer version. The setting of the dirty bit status for off-chip expired states is closely integrated with space reclamation operations. During space reclamation, the allocator checks the dirty bit status of the target on-chip data block. When the dirty bit status is off-chip expired, the allocator writes the data of the target on-chip data block back to off-chip memory. If the dirty bit status is not off-chip expired, there is no need to write back, and the data can be released directly. Therefore, the mechanism of updating off-chip storage data to on-chip expired state when it is modified, together with the mechanism of updating on-chip storage data to off-chip expired state when it is modified, constitutes a complete dirty bit state management closed loop. This ensures that data consistency can be accurately reflected when changes occur at the data source, whether it is located on-chip or off-chip. This effectively avoids calculation errors caused by outdated data and improves the reliability of the allocator.

[0115] Understandably, the dirty bit status is used to indicate data consistency between SPM and DRAM, including three states: consistent, on-chip expired, and off-chip expired. The memory mapping records the correspondence between on-chip data blocks and DRAM addresses, established and maintained by software during allocation. When the dirty bit status of an on-chip data block is either on-chip expired or off-chip expired, the allocator executes the corresponding flushToDram or flushToSpm operation. Specifically, when there is an on-chip data block with a dirty bit status of on-chip expired, data is loaded from off-chip storage to on-chip storage based on the memory mapping, and the dirty bit status of the on-chip data block is updated to consistent; when there is an on-chip data block with a dirty bit status of off-chip expired, the data of the on-chip data block is written back to off-chip storage based on the memory mapping, and the dirty bit status of the on-chip data block is updated to consistent.

[0116] In one embodiment, when accessing an on-chip data block whose dirty bit state is expired on-chip, directly reading the SPM would lead to a calculation error. Therefore, based on the storage mapping relationship, the corresponding address of the on-chip data block in the off-chip DRAM is first determined. Then, through a DMA controller or data copy engine, the latest data in the off-chip memory is loaded into the on-chip memory. After the data loading is complete, the dirty bit state of the on-chip data block is updated to a consistent state, indicating that the data in the on-chip memory and off-chip memory are synchronized. This loading operation is only performed when the on-chip data block is in an expired state, avoiding unnecessary data transfer operations and saving bandwidth and power consumption. The on-chip expired state is typically set when the data corresponding to the on-chip data block in the off-chip memory is modified externally, for example, by another computing unit or DMA controller performing a write operation on the off-chip memory. Furthermore, when a new free block is allocated in the on-chip memory, the dirty bit state of that free block can also be initialized to an expired state, indicating that the data block has not yet loaded valid data from the off-chip memory. For example, if a computing unit needs to access on-chip data block A of SPM, and the dirty bit of on-chip data block A is in an on-chip expired state, the allocator obtains the DRAM address 0x1000 based on the memory mapping relationship, and the instruction DMA loads the data from 0x1000 to on-chip data block A of SPM. After completion, the dirty bit state is updated to a consistent state, so that the computing unit can safely access it.

[0117] In one embodiment, when an on-chip data block has a dirty bit status indicating off-chip expiration, failure to write it back before space reclamation will result in the loss of dirty data. Therefore, firstly, based on the memory mapping relationship, the target address of the on-chip data block in DRAM is determined; then, the latest data in SPM is written back to DRAM via the DMA controller or memory instructions; after the write-back is completed, the dirty bit status is updated to a consistent state, indicating that the two layers of data have been synchronized. The write-back operation is triggered during space reclamation or active synchronization, i.e., it is performed on data blocks with a dirty bit status indicating off-chip expiration, while other states are skipped, avoiding unnecessary DRAM writes. The data write-back operation works in conjunction with the basic space reclamation operation to ensure that obsolete dirty data is not lost, and also supports actively writing the results back to DRAM after the computation task is completed. For example, if the computing unit modifies the on-chip data block B of SPM and the dirty bit is set to an off-chip expired state, when the on-chip data block needs to be reclaimed, the allocator checks the dirty bit state, finds the DRAM address based on the storage mapping relationship, writes the data of on-chip data block B back to DRAM through DMA, updates the dirty bit state to a consistent state, and then releases the SPM space, ensuring data safe synchronization.

[0118] Therefore, the data loading and write-back operations, combined with the data block allocation and state maintenance mechanisms in the above cache management method, form a complete data flow path, enabling on-chip storage to flexibly manage the data lifecycle and ensure that the latest version of data can be provided to the computing unit at any time.

[0119] In computing chips, the lifecycle of on-chip data blocks in on-chip memory is often closely related to asynchronous operations such as compute unit kernel execution and DMA data transfer. When a release operation is called, the data block may still be accessed by an asynchronous operation that has not yet been completed. If the SPM space occupied by the data block is reclaimed immediately, the space may be reallocated to other requests, causing ongoing asynchronous operations to access space that has already been reallocated, leading to data access errors or allocator crashes. In addition, if the data block is reclaimed too early, the data needs to be reloaded from off-chip memory, increasing unnecessary reload overhead. Therefore, this application proposes a lazy release block state, decoupling the logical release of data blocks from the actual reclamation of physical space, delaying physical reclamation until all associated asynchronous operations are completed. Specifically, in response to a lazy release request for an on-chip data block, the block state of the on-chip data block is updated to an idle state; when the associated event corresponding to the on-chip data block with an idle state is completed, the storage space occupied by the on-chip data block with an idle state is released, where the associated event represents a data access event, data transfer event, or compute unit execution event related to the on-chip data block.

[0120] A lazy release request is an instruction or signal initiated by software or hardware, instructing the allocator to logically release a block of on-chip data. The purpose of this request is to inform the allocator that the data block is no longer actively used in the current application logic, but its physical storage space should not be immediately reclaimed. For example, this request could be a specific API function, such as `free()`, provided by the application through a memory management library call. Upon receiving a lazy release request, the allocator changes the block status of the on-chip data block from its current active or locked state to an idle state. This idle state indicates that the data block has been logically released, but its physical storage space is still reserved and has not yet been reclaimed and reallocated. For example, the allocator can modify the block status field corresponding to the on-chip data block to set a specific code representing the idle state.

[0121] Among these, associated events are data access events, data movement events, or computation unit execution events related to the on-chip data block. Data access events can refer to read and write operations performed by a computation unit on the on-chip data block; for example, a computation core is reading input data from the data block or writing computation results to it. Data movement events can refer to the DMA controller moving data from external storage to the on-chip data block, or vice versa. Computation unit execution events can refer to a computation unit using the on-chip data block as its input or output buffer for computation tasks. After confirming all associated events, the allocator marks the physical storage space occupied by the on-chip data block as available and adds it to the free space pool available for subsequent allocation requests. Simultaneously, it can add the encoding of the on-chip data block to the free block list. For example, the allocator can add the starting address and size of the data block to the free block list of on-chip storage; or, through the memory management unit, unmap the physical address of the data block, making it available for reallocation.

[0122] In one specific implementation, suppose a computing unit is executing a computational task that requires reading data from on-chip data block A. After the computing unit starts the computational task, the application may immediately issue a lazy release request for on-chip data block A. Upon receiving this request, the block manager updates the block status of on-chip data block A from active to idle. However, the physical storage space occupied by on-chip data block A is not immediately reclaimed. The allocator maintains an event queue or status register to track the completion of the computing unit's access to on-chip data block A. When the computing unit completes all data reading and processing of on-chip data block A and issues a task completion signal, the allocator further checks for any other incomplete data transfer or data access events related to on-chip data block A. Once it is confirmed that all associated events have been completed, the block manager adds the physical storage space occupied by on-chip data block A to the list of free blocks in on-chip storage, making it reusable for subsequent data allocation requests.

[0123] In other words, when the allocator receives a lazy release request for an on-chip data block, it does not immediately reclaim the physical storage space it occupies. Instead, it updates the block's status to free. In this state, the data block is logically marked as releaseable, but its data content and physical space remain in the on-chip storage. The allocator continuously monitors asynchronous events associated with the on-chip data block, such as data access events, data transfer events, or compute unit execution events. Once all these associated events are confirmed, the allocator releases the physical storage space occupied by the on-chip data block and returns it to the free space pool for subsequent data allocation requests. Therefore, by decoupling the logical release of on-chip data blocks from the actual reclamation of physical space, the problem of safe reclamation of on-chip storage space in asynchronous operation scenarios is solved.

[0124] In practical applications, certain critical data blocks (such as deep learning model weights) need to reside on-chip storage for extended periods to ensure frequent access. Therefore, the block state of these critical data blocks is elevated from an active state to an eviction-immune state, preventing them from being evicted during space reclamation operations. Specifically, in response to a locking request for an on-chip data block, its block state is updated to a locked state, indicating that the on-chip data block is locked in on-chip storage; in response to an unlocking request for an on-chip data block, the block state of the locked on-chip data block is restored to an active state.

[0125] A lock request is an instruction issued by software or a specific hardware module, instructing the allocator to mark a particular on-chip data block as non-reclaimable. This request can be triggered by an application calling a specific API function, or automatically issued by the allocator / runtime environment according to a preset strategy. Upon receiving a lock request, the allocator modifies the metadata corresponding to the on-chip data block, changing the value of the block status field from the current active state or another state to the locked state. The locked state is a special block state indicating that the on-chip data block is fixed in on-chip storage and will not be selected for reclamation. This means that even if on-chip storage faces space pressure and needs reclamation, data blocks in the locked state will be excluded from the reclamation candidate set, ensuring their continued residence in on-chip storage. An unlock request, corresponding to a lock request, is an instruction used to release the lock on a specific on-chip data block. Similar to lock requests, unlock requests can also be triggered by software API calls or internal allocator mechanisms. Their purpose is to allow previously locked data blocks to re-participate in the space reclamation mechanism so that they can be reclaimed when needed, freeing up on-chip storage space. Upon receiving an unlock request, the allocator changes the block status of the target on-chip data block from locked to active. An active status indicates that the data block is currently in use or may be in use, and can be considered for space reclamation operations when on-chip storage space is insufficient.

[0126] Therefore, by introducing a locking and unlocking mechanism, critical data blocks can be explicitly locked in on-chip storage, thus becoming immune to space reclamation operations. This solves the problem of critical data blocks potentially being mistakenly evicted when on-chip storage space is insufficient, and avoids the frequent writing of critical data from on-chip storage back to off-chip storage and then reloading it back to on-chip storage, reducing bandwidth overhead and data transmission latency in off-chip storage. Simultaneously, because critical data blocks can reside stably in on-chip storage for extended periods, computing units can continuously access this data with high efficiency, avoiding computational interruptions and performance fluctuations caused by data loading, thereby improving overall computing efficiency and reducing allocator power consumption.

[0127] It should be noted that the space reclamation operation updates on-chip data blocks to an obsolete state to release storage space. An obsolete on-chip data block indicates that after the space reclamation operation, the SPM physical space has been released, the data has been written back to off-chip storage, but the storage mapping relationship is retained. Therefore, when it is necessary to access an on-chip data block with an obsolete state, the allocator will load the data from off-chip storage to on-chip storage based on the storage mapping relationship, restore the block state of the on-chip data block from obsolete to active, and update the dirty bit state to a consistent state.

[0128] Specifically, when a computing unit needs to access an on-chip data block whose block status is obsolete—that is, when the computing unit attempts to read or write an on-chip data block whose block status is currently obsolete—the allocator uses a pre-reserved storage mapping relationship to locate the corresponding location of the data block in off-chip storage. This storage mapping relationship is not deleted when the data block is obsolete but is retained as a key basis for data recovery. Subsequently, the allocator reads the data block's data from off-chip storage and loads it into an available space in on-chip storage through a data transfer mechanism. After the data is successfully loaded into on-chip storage, the allocator updates the on-chip data block's block status from obsolete to active. The active status indicates that the data block now holds valid data in on-chip storage and is available for normal read and write operations by the computing unit, allowing the computing unit to access, modify, or obsolete the data block again.

[0129] During block state recovery, the allocator updates the dirty bit status of the on-chip data block to a consistent state. A consistent state indicates that the data in on-chip storage is completely synchronized with the corresponding data in off-chip storage. Since the data has just been loaded from off-chip storage, the two contents are identical at this point, and marking it as consistent helps with subsequent cache consistency management. For example, if the data has not been modified during subsequent eviction, there is no need to write it back to off-chip storage, thus saving bandwidth.

[0130] Therefore, when a computing unit needs to access on-chip data blocks that have been evicted by space reclamation operations, the method provided in this application can quickly and accurately load the required data from off-chip storage to on-chip storage based on the retained storage mapping relationship. Simultaneously, by restoring the block state of the on-chip data block from an evicted state to an active state and updating its dirty bit state to a consistent state, the availability of data in on-chip storage and its consistency with off-chip storage are ensured. When space is scarce, on-chip space is freed up by evicting inactive data to meet new allocation requests; when needed, data can be seamlessly reloaded and activated, allowing for flexible utilization of on-chip storage space, effectively improving the effective utilization rate of on-chip storage, avoiding access interruptions or data loss due to data evicting, thereby ensuring the continuity of data access and the overall performance of the allocator. Compared to the traditional approach of deleting the mapping relationship upon evicting, this application retains the storage mapping relationship, avoiding the overhead of rebuilding the storage mapping relationship, ensuring the correctness of data loading, and forming a complete closed loop for data block lifecycle management, including allocation, use, evicting, recovery, and reuse.

[0131] The technical solution of this application will be described in detail below with reference to specific embodiments. It should be noted that these embodiments are only used to explain this application and are not intended to limit the scope of protection of this application. Parameter settings, data sizes, and values ​​are merely examples and do not constitute a limitation on this application.

[0132] Example 1

[0133] In deep learning inference tasks, model weights need to reside in the SPM (Structured Memory Manager) for an extended period to reduce latency caused by repeated loading from DRAM. Intermediate activation data, however, is only needed in the current computation layer and can be released promptly after computation for reuse by the next layer. Each computational core has a 1MB SPM capacity and comprises four computational units. Therefore, the weight and activation tensors need to be moved from DRAM to the SPM. The weight tensor is 128KB in size and accessed in shared mode; the activation tensor is 64KB in size and accessed in independent mode.

[0134] For weight tensors, since it's a shared mode, they can be directly loaded from DRAM to SPM. After loading, the application calls the `spm_pin()` interface to raise the block status of the on-chip data block storing the weight tensor from active (USING) to locked (PINNED). Once locked, the on-chip data block storing the weight tensor will be skipped in subsequent space reclamation operations and will not be selected for eviction, thus ensuring that the weight tensor continues to reside in SPM without needing to be repeatedly loaded from DRAM.

[0135] For the active tensor, the application initiates a data allocation request. Since the access mode is independent, the allocator rounds up the active tensor size to the nearest 256 bytes to obtain the partition size for each compute unit, which is 64KB. Then, the partition size is multiplied by the number of compute units to obtain 256KB of data space to be allocated. The allocator searches for free blocks in the SPM. If enough contiguous free blocks exist, they are allocated directly; otherwise, space reclamation or defragmentation is triggered. After allocation, the block status of each active data block is set to active (USING), and the dirty bit status is set to on-chip expired (MEM_DIRTY_DRAM), indicating that the SPM data is uninitialized and needs to be loaded from DRAM. The compute unit performs calculations on the active tensor. Since the runtime cannot automatically detect kernel write operations, after the calculation is complete, the application calls the `spm_mark_dirty()` interface to explicitly set the dirty bit status of the on-chip data block storing the active tensor to off-chip expired, clearly indicating that the SPM data has been modified and the corresponding data in DRAM is outdated. Therefore, the application calls the `flushToDram` operation to write the active tensor, whose dirty bit status is off-chip expired, back to DRAM. After the write-back is complete, the allocator updates the dirty bit status to a consistent state, indicating that the SPM and DRAM data are synchronized. Then, the application calls the `spm_free()` interface to release the on-chip data block storing the active tensor. The allocator updates its block status from active to free, but does not reclaim physical space immediately. It waits for the GPU events associated with the data block, such as the kernel execution, to complete and release the SPM space, ensuring that ongoing asynchronous operations will not access incorrect data due to space reallocation. When the next round of inference begins, new data allocation requests for active tensors can directly reuse the released SPM space. If, after multiple rounds of inference, the SPM is severely fragmented (i.e., the total free capacity is sufficient but there are no contiguous free blocks), the allocator will automatically trigger a defragmentation operation. This involves compactly arranging active and locked blocks, merging gaps into contiguous free blocks, and restoring contiguous allocation capability.

[0136] Based on the above examples, weighted data can reside in the SPM for an extended period using a locking mechanism, eliminating the need for repeated loading from DRAM, reducing data transfer latency and DRAM bandwidth consumption. Activation data is promptly written back to DRAM after calculation, releasing SPM space and achieving cyclical utilization of SPM space, ensuring efficient use of limited storage resources. Therefore, by explicitly setting the dirty bit status and writing back on demand, unnecessary data transfers are avoided, and write-back operations are performed when data is modified, saving bandwidth and power consumption.

[0137] Example 2

[0138] In long-running training or inference tasks, tensors of different sizes are allocated and released alternately, resulting in multiple small free blocks scattered throughout the SPM (Single-Planetary Memory). Although the total free capacity is sufficient, it cannot meet large consecutive allocation requests. Specifically, the SPM has a total capacity of 1MB, with 5 currently active on-chip data blocks occupying a total of 600KB, and 3 currently idle data blocks of 50KB, 30KB, and 20KB in size, for a total free capacity of 100KB. At this point, a new data allocation request indicates a data size of 80KB.

[0139] After receiving an 80KB data allocation request, the allocator traverses the list of free blocks and finds that the total free capacity of 100KB is greater than the requested size of 80KB, but the largest consecutive free block of 50KB is less than the requested size of 80KB. Therefore, it performs a defragmentation operation.

[0140] Specifically, the allocator collects all data blocks in the SPM (Storage Management Platform) that are in active or locked states, resulting in five active on-chip data blocks. These are then sorted by their physical addresses in the SPM from smallest to largest. Following this sorting order, the five active on-chip data blocks are sequentially and compactly arranged into the starting address region of the SPM. Before moving each data block, it's necessary to determine if its dirty bit status is off-chip expired and requires movement. If the dirty bit status is off-chip expired, a `flushToDram` operation must be performed to write the data back to DRAM before moving the data within the SPM, ensuring no changes are lost during the move. If the dirty bit status is consistent or on-chip expired, the data can be moved directly within the SPM without writing back. After the move is complete, the allocator updates the SPM physical address of each data block in the block storage mapping, ensuring that subsequent accesses can locate the data using the correct address. After all five active on-chip data blocks are compactly arranged in the SPM header, the three idle on-chip data blocks that were originally located in the gaps between blocks and at the tail are merged into a large contiguous free area, totaling 400KB of contiguous space. The allocator adds this contiguous space to the free block list and resets the free block list. From the consolidated 400KB contiguous free block, the allocator allocates 80KB to the data allocation request, and the allocation is successful.

[0141] Based on the above example, after the defragmentation operation, the maximum contiguous free block size increased from 50KB to 400KB, restoring SPM's contiguous allocation capability and avoiding allocation failures caused by fragmentation.

[0142] Example 3

[0143] When active blocks in the SPM occupy most of the space, and new allocation requests require more space, the allocator needs to evict some active blocks to free up space. The evicted data may be needed again later, in which case it can be reloaded from DRAM through a recovery operation.

[0144] The allocator receives a new data allocation request, but there are no free blocks in the current SPM (Site-Mounted Physical Block) that can accommodate the data to be allocated, triggering an eviction policy, i.e., performing a space reclamation operation. The allocator selects one or more target eviction blocks from all active on-chip data blocks according to a preset eviction policy, such as the Least Recently Used (LRU) policy. For each selected target eviction block, the allocator checks the dirty bit status. If the dirty bit status is off-chip expired, the allocator performs a flushToDram operation to write the dirty data back to DRAM and update the dirty bit to a consistent status, ensuring no data loss. If the dirty bit status is consistent or on-chip expired, no write-back is needed, and the block can be released directly. After completing the necessary write-back, the allocator releases the SPM physical space occupied by the target eviction block and updates the block status from active to eviction. The eviction status indicates that the data block has been evicted from the SPM, but its storage mapping relationship is retained, providing a basis for subsequent recovery. The released SPM space is added to the free block list for new allocation requests. The allocator allocates the required space from the newly freed blocks, completing the allocation process. If the evicted data needs to be accessed again later, such as when recalculating the gradient of a layer during deep learning training, the application can call the `flushToSpm` operation to reload the data from DRAM to SPM based on the retained storage mapping. After loading, the allocator restores the block state from evicted to active, updates the dirty bit state to consistent, and indicates that the SPM and DRAM data are synchronized. At this point, the evicted data block is restored and can be used for subsequent computations.

[0145] Based on the above example, through dirty bit tracking, only dirty data in the SPM is written back during eviction. Blocks in a consistent state or on-chip expired state are directly released, avoiding unnecessary data transfer and saving DRAM bandwidth and power consumption. Evicted blocks can be restored on demand via the flushToSpm operation, achieving flexible utilization of SPM space. They are released when space is scarce and reloaded when needed, allowing the limited SPM capacity to support data access demands far exceeding its size. Therefore, the combination of eviction status and restoration mechanisms enables data blocks to flow flexibly between SPM and DRAM, ensuring space utilization and data access continuity.

[0146] This application also provides a cache management device, including at least one processor and a memory communicatively connected to the at least one processor; wherein the processor and the memory are communicatively connected via a data bus, the memory stores a program, and the program is executed by the at least one processor to enable the at least one processor to implement the method as described in any of the above embodiments of this application when executing instructions.

[0147] The following is combined Figure 6 The hardware structure of the cache management device is described in detail. The cache management device includes: a processor 610, a memory 620, an input / output interface 630, a communication interface 640, and a bus 650.

[0148] The processor 610 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0149] The memory 620 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 620 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610 using the cache management method of the embodiments of this application.

[0150] The input / output interface 630 is used to implement information input and output;

[0151] The communication interface 640 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.).

[0152] Bus 650 transmits information between various components of the device (e.g., processor 610, memory 620, input / output interface 630, and communication interface 640);

[0153] The processor 610, memory 620, input / output interface 630 and communication interface 640 are connected to each other within the device via bus 650.

[0154] This application proposes a computing chip including the cache management device of the above embodiments. It is worth noting that, since the computing chip of this application includes the cache management device of any of the above embodiments, the specific implementation methods and technical effects of the computing chip of this application can be referred to the specific implementation methods and technical effects of the cache management device of any of the above embodiments.

[0155] It's worth noting that computing chips can be either in-memory computing chips or chips with a non-in-memory computing architecture. Computing chips include on-chip storage and connect to external off-chip storage. Off-chip storage is a large-capacity storage layer, while on-chip storage is a small-capacity, low-latency, and high-bandwidth on-chip fast storage, with significantly smaller capacity and latency than off-chip storage. A single chip contains multiple computing units, such as four computing units, which share a single piece of on-chip storage. Each piece of on-chip storage is owned by one chip and shared by the multiple computing units within that chip. To accelerate data access using on-chip storage, data needs to be loaded from off-chip storage to on-chip storage, and after computation, the modified data is written back to off-chip storage.

[0156] This application proposes an electronic device including the computing chip of the above embodiments. It is worth noting that, since the electronic device of this application includes the computing chip of any of the above embodiments, the specific implementation and technical effects of the electronic device of this application can be derived by referring to the specific implementation and technical effects of the computing chip of any of the above embodiments.

[0157] This application also provides a computer-readable storage medium for storing a computer program for executing the cache management methods of the foregoing embodiments.

[0158] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the cache management method described above.

[0159] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0160] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0161] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0165] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0166] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A cache management method, characterized in that, Applied to a computing chip including on-chip storage, the computing chip being connected to off-chip storage, the method includes: Maintain the dirty bit state and block state of each on-chip data block in the on-chip storage, wherein the dirty bit state is used to indicate data consistency between the on-chip storage and the off-chip storage, and the block state is used to indicate the lifecycle stage of the on-chip data block; Receive a data allocation request for the on-chip storage, and determine the data space to be allocated based on the data allocation request; If there are no free blocks in the on-chip storage that satisfy the data space to be allocated, a space reclamation operation is performed. The space reclamation operation includes: selecting at least one on-chip data block whose block status is active as the target on-chip data block, releasing the storage space occupied by the target on-chip data block, and updating the block status of the target on-chip data block to the evicted state. When the dirty bit status of the target on-chip data block is the off-chip expired state, the data of the target on-chip data block is written back to the off-chip storage. The off-chip expired state indicates that the data in the on-chip storage has been modified and has not been synchronized to the off-chip storage. If there is a free block in the on-chip storage that satisfies the data space to be allocated, the free block is allocated to the data allocation request, a storage mapping relationship is established between the free block and the corresponding address in the off-chip storage, the block status of the free block is updated to active status and the dirty bit status is updated to on-chip expired status, wherein the on-chip expired status indicates that the data in the off-chip storage has been modified and needs to be reloaded to the on-chip storage.

2. The cache management method according to claim 1, characterized in that, If the on-chip storage has a free block that satisfies the data space to be allocated, allocating the free block to the data allocation request includes: If there are consecutive free blocks in the on-chip storage that satisfy the data space to be allocated, the consecutive free blocks are directly allocated to the data allocation request.

3. The cache management method according to claim 1, characterized in that, The step of allocating the free block to the data allocation request if there is a free block in the on-chip storage that satisfies the data space to be allocated, further includes: If the total free capacity of the on-chip storage satisfies the data space to be allocated, and there are no consecutive free blocks that satisfy the data space to be allocated, a defragmentation operation is performed on the on-chip storage to obtain consecutive free blocks that satisfy the data space to be allocated, and the consecutive free blocks are allocated to the data allocation request. The defragmentation operation includes: On-chip data blocks that are in an active state and / or a locked state are designated as data blocks to be processed. The data blocks to be organized are compactly arranged in address order on the on-chip storage, and the storage mapping relationship is updated. Organize the continuous free areas formed by compact arrangement into at least one continuous free block.

4. The cache management method according to claim 3, characterized in that, Before the step of compactly arranging the data blocks to be organized in address order on the on-chip storage, the method further includes: The data of the data block to be processed, whose current dirty status is out-of-chip expired, is written back from the on-chip storage to the off-chip storage, and the dirty status of the data block to be processed is updated to a consistent status, which means that the data of the on-chip data block is consistent with the data in the off-chip storage.

5. The cache management method according to claim 1, characterized in that, The computing chip further includes multiple computing units, and the step of determining the data space to be allocated based on the data allocation request includes: If the access mode of the data allocation request is independent mode, the number of computing units and the size of the data to be allocated indicated by the data allocation request are multiplied to obtain the data space to be allocated; If the access mode of the data allocation request is the shared mode, the size of the data to be allocated indicated by the data allocation request is aligned according to the preset alignment bytes, and the aligned data size is used as the data space to be allocated.

6. The cache management method according to claim 1, characterized in that, The organ location status includes consistent status, on-chip expired status, and off-chip expired status; When the data corresponding to the on-chip data block stored off-chip is modified, the dirty bit status of the on-chip data block is updated to the on-chip expired status.

7. The cache management method according to claim 6, characterized in that, When the data stored on the chip is modified, the dirty bit status of the on-chip data block corresponding to the modified data is updated to the off-chip expired status.

8. The cache management method according to claim 6, characterized in that, When there is an on-chip data block with a dirty bit state of on-chip expiration, data is loaded from the off-chip storage to the on-chip storage based on the storage mapping relationship, and the dirty bit state of the on-chip data block is updated to a consistent state.

9. The cache management method according to claim 1, characterized in that, When there is an on-chip data block with a dirty bit state that is out of chip range, the data of the on-chip data block is written back to the off-chip storage based on the storage mapping relationship, and the dirty bit state of the on-chip data block is updated to a consistent state.

10. The cache management method according to claim 1, characterized in that, Also includes: In response to a lazy release request for the on-chip data block, the block status of the on-chip data block is updated to idle. When the associated event corresponding to an on-chip data block in an idle state is completed, the storage space occupied by the on-chip data block in an idle state is released. The associated event represents a data access event, data transfer event, or computing unit execution event related to the on-chip data block.

11. The cache management method according to claim 1, characterized in that, Also includes: In response to a lock request for the on-chip data block, the block status of the on-chip data block is updated to a locked state, which indicates that the on-chip data block is locked in the on-chip storage; In response to an unlock request for the on-chip data block, the block state of the on-chip data block that was in a locked state is restored to an active state.

12. The cache management method according to claim 1, characterized in that, Also includes: When it is necessary to access an on-chip data block whose block status is obsolete, data is loaded from the off-chip storage to the on-chip storage based on the storage mapping relationship, and the block status of the on-chip data block is restored from obsolete to active, and the dirty bit status is updated to consistent. The obsolete on-chip data block represents a free block generated after the space reclamation operation.

13. The cache management method according to claim 1, characterized in that, The dirty bit status and block status of the on-chip data blocks are maintained and updated through explicit settings via a predefined software interface.

14. A cache management device, characterized in that, The system includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. The program is executed by the processor to implement the cache management method as described in any one of claims 1 to 13.

15. A computing chip, characterized in that, It includes on-chip storage, multiple computing units, and a cache management device as described in claim 14, wherein the computing chip is connected to off-chip storage.

16. An electronic device, characterized in that, Includes the computing chip as described in claim 15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the cache management method as described in any one of claims 1 to 13.