Request management methods, devices, electronic devices and storage media

Abstract

This invention relates to the field of computer technology, providing a request management method, apparatus, electronic device, and storage medium. The method includes: upon detecting a cache miss in a data access request, checking whether an incomplete request entry corresponding to a first address exists in a preset request management structure based on a first address carried by the data access request. The preset request management structure is a pre-constructed multi-way set-associative cache structure for grouping and managing cache miss requests. If the incomplete request entry exists in the preset request management structure, adding the data access request to the waiting queue of the incomplete request entry. This invention, by employing a multi-way set-associative cache structure for grouping and managing cache miss requests, enables parallel lookups of cache miss requests, significantly improving the processing efficiency of cache miss requests.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more particularly to a request management method, apparatus, electronic device, and storage medium. Background Technology

[0002] In modern computer systems, some systems employ a FIFO (First In First Out) queue structure to manage cache miss requests. The hardware scheduler monitors the cache line loading status of the request at the head of the queue to process requests in the queue sequentially. However, current designs suffer from significant performance bottlenecks. This is primarily due to the inherent characteristics of memory systems, which make request response times unpredictable. The strict FIFO processing mode causes blocking of requests at the head of the queue, making out-of-order processing impossible and hindering the merging of cache miss requests with the same address, thus reducing overall memory access parallelism. This single-queue-based sequential processing mechanism cannot fully utilize the parallel capabilities of modern memory systems, resulting in a significant decrease in request processing efficiency. Summary of the Invention

[0003] This invention provides a request management method, apparatus, electronic device, and storage medium to address the shortcomings of low efficiency in handling cache miss requests in related technologies.

[0004] This invention provides a request management method, comprising:

[0005] When a cache miss is detected in a data access request, the system checks whether there is an incomplete request entry corresponding to the first address in the preset request management structure based on the first address carried in the data access request. The preset request management structure is a pre-built multi-way group-associative cache structure used to group and manage cache miss requests.

[0006] If an incomplete request entry exists in the preset request management structure, the data access request is added to the waiting queue of the incomplete request entry.

[0007] According to a request management method provided by the present invention, adding the data access request to the waiting queue of the incomplete request entries includes:

[0008] Obtain the all-zero flag of the incomplete request entry;

[0009] If the all-zero flag is true, return all-zero data directly to the data access request;

[0010] If the all-zero flag is false, the data access request is added to the waiting queue of the incomplete request entry.

[0011] According to a request management method provided by the present invention, the preset request management structure includes multiple groups, each group including multiple entries, and the step of checking whether there is an incomplete request entry corresponding to the first address in the preset request management structure based on the first address carried by the data access request includes:

[0012] Based on the first address, determine the group index and entry tag;

[0013] Based on the group index, a target group is determined from multiple groups in the preset request management structure, and the entry tag is matched with the tag of each entry in the target group;

[0014] If a match is found, the matched entry will be identified as the incomplete request entry.

[0015] If the matching fails, it is determined that there is no incomplete request entry in the preset request management structure.

[0016] According to a request management method provided by the present invention, it further includes:

[0017] If there is no incomplete request entry in the preset request management structure, check if there is a free entry in the target group of the preset request management structure. The target group is determined based on the first address carried by the data access request, and the free entry is determined based on the validity flag bit of each entry in the target group.

[0018] If a free entry exists in the target group, the free entry will be assigned as an incomplete request entry corresponding to the first address.

[0019] According to a request management method provided by the present invention, the step of allocating the free entry as an incomplete request entry corresponding to the first address includes:

[0020] Store the entry tag carried in the first address into the tag flag bit of the free entry;

[0021] Set the validity flag of the free entry to true, and set the data return flag and all-zero flag of the free entry to false;

[0022] The data access request is added to the waiting queue of the idle entry, and a data request is initiated to the lower-level storage.

[0023] According to a request management method provided by the present invention, it further includes:

[0024] If no free entry exists in the target group, check if there is an all-zero entry in the target group of the preset request management structure. The all-zero entry is determined based on the all-zero flag bit of each entry in the target group.

[0025] If it exists, the all-zero entry is replaced so that the all-zero entry is assigned as the incomplete request entry corresponding to the first address;

[0026] If it does not exist, the data access request is determined to be blocked, and the process waits for the entry in the target group to be released.

[0027] According to a request management method provided by the present invention, it further includes:

[0028] After receiving the target data returned by the lower-level storage, based on the second address carried by the target data, the target request entry matching the second address is searched in the preset request management structure;

[0029] The target data is written to the cache, and the data return flag of the target request entry is set to true to trigger the execution of all requests in the waiting queue of the target request entry.

[0030] According to a request management method provided by the present invention, writing the target data into the cache includes:

[0031] If the target data is all zeros, set the all-zero flag of the target request entry to true;

[0032] If the target data is not all zeros, the target data is written to the cache.

[0033] According to a request management method provided by the present invention, the step of executing all requests in the waiting queue of the target request entry includes:

[0034] Each cycle, a request is taken from the waiting queue of the target request entry, and the target data is returned to the request, until the waiting queue of the target request entry is empty, at which point the target request entry is released.

[0035] The present invention also provides a request management device, comprising:

[0036] The checking unit is used to check whether there is an incomplete request entry corresponding to the first address in the preset request management structure when a cache miss is detected in a data access request, based on the first address carried by the data access request. The preset request management structure is a pre-built multi-way group-associative cache structure for grouping and managing cache miss requests.

[0037] The addition unit is used to add the data access request to the waiting queue of the incomplete request entry when there is an incomplete request entry in the preset request management structure.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the request management method as described above.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the request management method as described above.

[0040] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the request management method as described above.

[0041] The request management method, apparatus, electronic device, and storage medium provided by this invention, when a cache miss is detected in a data access request, checks whether a corresponding incomplete request entry exists in a preset request management structure based on the first address carried in the request. If it exists, the request can be added to the waiting queue of that entry. This means that when a request with the same address is detected, subsequent requests can be directly added to the waiting queue of existing requests, effectively merging cache miss requests with the same address. This eliminates the need to send multiple requests to lower-level storage for the same address, thus avoiding the overhead of repeatedly accessing lower-level storage and significantly reducing system bandwidth consumption and access latency. Furthermore, the preset request management structure is a pre-built multi-way set-associative cache structure. By using this set-associative cache structure instead of the traditional FIFO structure, group management and parallel lookup of cache miss requests can be achieved. This effectively avoids problems such as queue head blocking and inability to process out-of-order requests caused by sequential processing in the traditional FIFO structure, fully utilizing the parallel characteristics of the memory system and significantly improving the processing efficiency of cache miss requests. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the structure of the artificial intelligence chip provided by the present invention;

[0044] Figure 2This is a flowchart illustrating the request management method provided by the present invention;

[0045] Figure 3 This is a schematic diagram of a group in the preset request management structure provided by the present invention;

[0046] Figure 4 This is a flowchart illustrating the parallel cache miss request handling method based on a group-associative cache structure provided by the present invention.

[0047] Figure 5 This is a schematic diagram of the request management device provided by the present invention;

[0048] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] Figure 1 This is a schematic diagram of the structure of the artificial intelligence chip provided by the present invention, as shown below. Figure 1 As shown, the artificial intelligence chip 100 includes at least multiple streaming processing clusters (SPCs) 101 and memory 102. Each SPC includes multiple computing units 103 and a cluster bus interface (CBI) 105. The computing units 103 are used to execute corresponding computing tasks. Here, the memory 102 can be HBM (High Bandwidth Memory) or other types of memory.

[0051] Each computing unit 103 includes a Level 1 cache (L1 cache) 104, which is a temporary storage device with a smaller capacity than the main memory 102 but a faster data exchange speed. The AI ​​chip 100 also includes a Level 2 cache (L2 cache) 106, which is located outside the SPC. Its capacity is smaller than the main memory 102 but larger than the L1 cache, and its data exchange speed is faster than the main memory 102 but slower than the L1 cache.

[0052] When each computing unit 103 needs to read data, it searches for and retrieves the data in the following order: First, it checks the nearest L1 cache. If the data exists there, it is read directly, which is the fastest access method. If the data is not in the L1 cache, the system searches the L2 cache through the cluster bus interface 105. If the data is found in the L2 cache, it is read from the L2 cache. If the required data is not found in the L2 cache either, the system finally reads the data from memory 102 (such as HBM), which is the slowest access level but has the largest storage capacity.

[0053] It should be noted that, in addition to the structures described above, the artificial intelligence chip 100 may also include other structures, and this invention does not specifically limit its inclusion. The artificial intelligence chip 100 may be a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), a TPU (Tensor Processing Unit), a CPU (Central Processing Unit), etc. For ease of understanding of the technical solution provided by this invention, the following description uses a GPU as an example of the artificial intelligence chip 100.

[0054] During the data reading process of the aforementioned computing unit, a cache miss occurs when the required data is not in the L1 cache. In this case, the data needs to be loaded from lower-level storage (such as L2 cache, memory, etc.). Currently, the L1 cache typically uses a FIFO queue-based structure to manage these cache miss requests, ensuring that requests are processed in the order they arrive. When a cache miss occurs, the relevant request is placed in the FIFO queue, and the hardware scheduler processes these requests sequentially according to the queue order. Here, the hardware scheduler is the hardware component responsible for managing and scheduling cache miss requests. It continuously monitors the cache line loading status of the requests at the head of the FIFO queue and processes these miss requests strictly according to the order in which they entered the queue, thus achieving an in-order processing mechanism. That is, the scheduler will only process the next request in the queue after the cache line corresponding to the request at the head of the queue has been successfully loaded into the L1 cache. This mechanism guarantees the sequential processing of cache miss requests.

[0055] However, this seemingly orderly processing method has revealed many flaws in actual operation, becoming a key bottleneck restricting the improvement of system performance, specifically in the following aspects:

[0056] (1) Uncertainty of request response time: The characteristics of the memory system determine that the response time of different requests varies greatly. For example, some requests may need to access distant main memory areas, or the latency may increase due to memory access conflicts. In the traditional FIFO processing mode, even if the response time of subsequent requests entering the queue is short, they must wait for the long-delayed requests at the head of the queue to be processed before they can be processed, which undoubtedly increases the average waiting time of the overall requests.

[0057] (2) Limited processing efficiency:

[0058] The strict FIFO processing mode leads to queue head blocking: once the request at the head of the queue is blocked for some reason (such as memory access latency, resource contention, etc.), all subsequent requests will be forced to wait. Even if the data of these subsequent requests may be ready (e.g., hit in L2 cache), they cannot be processed in time, which seriously affects processing efficiency.

[0059] Even if subsequent requests for data are ready, out-of-order processing cannot be achieved: modern memory systems themselves have a certain parallel processing capability and can theoretically handle multiple independent requests simultaneously. However, traditional sequential processing mechanisms limit the utilization of this parallel characteristic, making it impossible to fully utilize the parallel characteristics of the memory system and further reducing processing efficiency.

[0060] Difficulty in merging cache misses for the same address: In practical applications, multiple data access requests targeting the same address may all miss the L1 cache. In traditional FIFO queue structures, it is difficult to effectively scan the queue for existing misses, making it impossible for the system to merge these requests with the same address. This results in multiple duplicate requests for the same address, increasing memory bandwidth usage and system overhead, and reducing processing efficiency.

[0061] Reducing overall memory access parallelism: Modern memory systems possess a certain degree of parallel processing capability, capable of handling multiple data access requests simultaneously. However, the traditional FIFO queue sequential processing mechanism limits the full realization of this parallelism. Since all requests must be processed in FIFO order, the parallel resources of the memory system cannot be fully utilized, leading to a reduction in overall memory access parallelism. This results in numerous idle cycles for computing units while waiting for data to load, severely hindering system performance improvements.

[0062] In this regard, the present invention is based on the following Figure 1The architecture diagram of the AI ​​chip shown provides a request management method. By redesigning the traditional FIFO queue structure into a multi-way set-associative cache structure, it achieves efficient group management and parallel lookup of cache miss requests. It effectively avoids the problems of queue head blocking, inability to process out-of-order requests, and difficulty in merging requests with the same address caused by sequential processing in the traditional FIFO queue structure. It can make full use of the parallel characteristics of the memory system and greatly improve the processing efficiency of cache miss requests, thereby overcoming the above defects.

[0063] It should be noted that the request management method provided by this invention is applicable to managing and processing all L1 cache miss requests, as well as all L2 cache miss requests, and also to managing and processing miss requests from other types of caches. This invention does not specifically limit its application in this regard. To facilitate understanding of the technical solution provided by this invention, the following description will take the management and processing of L1 cache miss requests as an example.

[0064] Figure 2 This is a flowchart illustrating the request management method provided by the present invention, as follows: Figure 2 As shown, the method includes:

[0065] Step 210: If a cache miss is detected in a data access request, check whether there is an incomplete request entry corresponding to the first address in the preset request management structure based on the first address carried in the data access request. The preset request management structure is a pre-built multi-way group-associative cache structure for grouping and managing cache miss requests.

[0066] It's important to note that in GPU applications, a data access request refers to a request sent by a computing unit to a cache (such as the L1 cache) in order to read data. This request contains the address information of the data to be accessed, and the caching system uses this address to locate the corresponding data. For example, when a computing unit in a GPU needs to read certain data, it will initiate a data access request to the L1 cache, hoping to quickly retrieve the required data from the L1 cache to improve the overall operating efficiency of the system.

[0067] A cache miss occurs when a computing unit issues a data access request, but the requested data is not stored in the cache (such as the L1 cache). In other words, the caching system searches for the data based on the address carried in the data access request, but cannot find the corresponding data. This situation is called a cache miss. For example, a computing unit accesses a memory address A through a data load instruction, hoping to read data from address A, but this data has not been loaded into the L1 cache before; in this case, an L1 cache miss occurs.

[0068] Specifically, when a caching system receives a data access request, it can employ a specific lookup mechanism to detect whether a cache miss has occurred. Specifically, the caching system searches the cache (such as the L1 cache) based on the first address carried in the data access request. The cache is typically divided into multiple cache lines, each storing a data block of a certain size and its corresponding address label. The caching system compares the label portion of the request address (i.e., the first address) with the address labels in the cache lines. If the label portion of the request address matches the address label of a cache line, and that cache line is valid (i.e., it stores valid data), then the cache hit occurs, and the data can be retrieved directly from that cache line. If the label portion of the request address does not match the address labels of any cache lines, or if the matching cache line is invalid, then a cache miss occurs.

[0069] Here, the first address carried in the aforementioned data access request refers to the memory address of the target data (i.e., the data to be retrieved) specified by the computing unit when initiating the data access request. In a computer system, data is stored in memory (such as HBM within a GPU), and each piece of data has its unique memory address. The computing unit uses this address to locate and access the required data. When the computing unit initiates a data access request, it includes this address information in the request so that the caching system can locate the data based on that address.

[0070] When a cache miss is detected for a data access request, it means that the required data is not in the cache and needs to be retrieved from lower-level storage (such as L2 cache, memory, etc.). Simultaneously, to facilitate subsequent processing of these cache miss requests, a structure is needed to store these requests. This way, once the lower-level storage returns the required data, these requests can be processed. To achieve efficient management of cache miss requests, this embodiment of the invention proposes a pre-built request management structure, which is a pre-constructed multi-way group-associative cache structure for grouping and managing cache miss requests. Here, a cache miss request refers to a data access request that has resulted in a cache miss.

[0071] As is understandable, a multi-way set-associative cache (MAC) is an organizational form in computer caching systems. In a MACC, the cache is divided into multiple groups, each containing multiple paths (or entries). For example, a default request management structure can be an 8-way MACC, which may include multiple groups, each containing 8 entries. When a data access request results in a cache miss, the default request management structure records relevant information for subsequent processing. The default request management structure resembles a table, where each entry represents an incomplete request, including address information, data return status, waiting queue, etc.

[0072] Specifically, the preset request management structure can group cache miss requests based on the addresses carried in the data access requests. When a cache miss is detected, the system checks the preset request management structure for an incomplete request entry corresponding to the first address carried in the request. Here, the caching system first locates the group in the preset request management structure based on the group index in the first address, then iterates through all entries in that group, comparing the tag of each entry with the tag in the first address. If they match, it indicates that an incomplete request entry corresponding to the first address exists in the preset request management structure; if they do not match, it continues to check the next entry until all entries in the group have been traversed.

[0073] The incomplete request entry corresponding to the first address mentioned above refers to an entry in the preset request management structure that stores information about incomplete requests associated with the first address carried in the data access request. This entry may include address information, data return status, waiting queue, and other relevant information. The address information is the information associated with the first address; the data return status indicates whether the data for this entry has been returned; the waiting queue is used to store other cache miss requests with the same address. When the data for this entry is retrieved and loaded into the cache, the requests in these waiting queues can be processed sequentially.

[0074] Step 220: If there are incomplete request entries in the preset request management structure, add the data access request to the waiting queue of the incomplete request entries.

[0075] It should be noted that when an incomplete request entry corresponding to the first address is detected in the preset request management structure, it indicates that at the current moment, other computing units or different threads within the same computing unit have already initiated access to the first address carried in the data access request, and that this request also resulted in a cache miss. Furthermore, the relevant information of this incomplete request has been recorded in the preset request management structure. This means that subsequent requests to this first address can share the processing progress of previous requests. For example, if the first request to this address has already initiated a data retrieval operation to lower-level storage (such as L2 cache, memory, etc.), subsequent requests do not need to repeatedly initiate requests to lower-level storage; they only need to wait for the data from the previous request to be returned.

[0076] Specifically, when an incomplete request entry corresponding to the first address is detected in the preset request management structure, the request can be directly added to the waiting queue of the incomplete request entry; if there is no incomplete request entry for the first address in the preset request management structure, a new entry is allocated to record the relevant information of the request.

[0077] For each incomplete request entry, the default request management structure maintains a waiting queue. This queue is a data structure used to store subsequent requests for the same address that have experienced a cache miss. When multiple requests for the same address all experience cache misses, these requests are added to the waiting queue of the incomplete request entry corresponding to that address, thus merging cache miss requests for the same address into a single entry. Once the data for that entry is retrieved and loaded into the cache, the requests in the waiting queue are processed sequentially, avoiding duplicate requests to lower-level storage and improving system efficiency.

[0078] For example, when a computing unit initiates a data access request for address A for the first time, and the request results in a cache miss, the caching system allocates a new entry in the preset request management structure, stores the address A of the request in that entry, and marks the request as incomplete. If other computing units subsequently initiate data access requests for address A and also experience cache misses, these requests will be added to the waiting queue of that entry, waiting for the data to be retrieved and processed.

[0079] Understandably, access to lower-level storage is relatively slow. If a request is sent to the lower-level storage every time a cache miss occurs, a significant amount of time will be wasted waiting for a response from the lower-level storage. By adding subsequent miss requests for the same address to a waiting queue, only one lower-level storage access is needed, greatly reducing the number of accesses to the lower-level storage and improving data access efficiency.

[0080] The method provided in this invention checks whether a corresponding incomplete request entry exists in a preset request management structure based on the first address carried in the request when a cache miss is detected. If it exists, the request can be added to the waiting queue of that entry. This means that when a request with the same address is detected, subsequent requests can be directly added to the waiting queue of existing requests, effectively merging cache miss requests with the same address. This avoids the overhead of repeatedly accessing lower-level storage, significantly reducing system bandwidth consumption and access latency. Furthermore, the preset request management structure is a pre-built multi-way group-associative cache structure. By using this group-associative cache structure instead of the traditional FIFO structure, group management and parallel lookup of cache miss requests can be achieved. This effectively avoids problems such as queue head blocking and inability to process out-of-order requests caused by sequential processing in the traditional FIFO structure, fully utilizing the parallel characteristics of the memory system and greatly improving the processing efficiency of cache miss requests.

[0081] Based on any of the above embodiments, the preset request management structure includes multiple groups, each group includes multiple entries, and each entry includes information such as a flag bit, an all-zero flag bit, a validity flag bit, a data return flag bit, and a waiting queue.

[0082] Specifically, the default request management structure is a multi-way group-associative cache structure, which can include multiple groups, each containing multiple entries. Here, the multiple groups in the default request management structure are set up for grouping and managing cache miss requests. It distributes all cache miss requests into different groups according to certain rules (such as address-based group indexes). Group management improves the efficiency of finding specific request entries. When it is necessary to find an incomplete request entry corresponding to a certain address, the system only needs to locate the specific group based on the group index calculated from the address, and then search within that group, without traversing the entire default request management structure.

[0083] The entries in each group are used to record information about specific cache miss requests. When multiple requests for different addresses experience cache misses, they are assigned to different entries; when multiple requests for the same address experience cache misses, they are added to the waiting queue of the same entry.

[0084] Figure 3 This is a schematic diagram of a group in the preset request management structure provided by the present invention, such as... Figure 3 As shown, the default request management structure can include multiple groups, each with 8 entries (e.g., entries 0 to 7, also known as way0 to way7). The "Nth group (also called set N)" in the diagram indicates that this group is the Nth group in the default request management structure, and the table in the diagram is a schematic diagram of this group. Each entry can include information such as a tag flag, an all-zero flag (isAllZero), a validity flag (valid), a data return flag (isReturned), and a waiting queue (pending requestfifo).

[0085] The tag is used to store the tag portion of the address information. In computer storage systems, addresses are typically divided into multiple parts, such as the tag portion (also called the entry tag portion), the set index portion, and the offset portion. For example, assuming a 32-bit address, the lower 6 bits are the offset portion, the middle 6 bits are the set index portion, and the remaining 20 bits are the tag portion. When the system processes a data access request that has a cache miss, it first extracts the set index portion and the tag portion based on the address of the request. Then, it finds the corresponding group based on the set index and compares the tag of each entry in that group with the tag portion extracted from the address to check if there is a matching tag, thereby determining whether there is an incomplete request entry for that address. It should be understood that the offset portion in the address is mainly used to locate the specific data in the cache line (such as the cache line in the L1 cache), and its correlation with the preset request management structure in this embodiment is not high, so it will not be described in detail here.

[0086] The validity flag (i.e., valid) indicates whether the entry is valid. For example, when valid=0, the entry is invalid, meaning it is idle and can be used by new requests; when valid=1, the entry is valid, meaning it is processing an incomplete request. It should be understood that when the system initializes the default request management structure, all entries have valid set to 0, indicating they are all idle. When a new data access request results in a cache miss, the system allocates an entry from the idle entries (valid=0) to record the relevant information of the request and sets the valid flag of that entry to 1. Once the request is processed, the system sets the valid flag of that entry to 0, returning it to an idle state.

[0087] The data return flag (i.e., isReturned) indicates whether the data for this entry has been returned. This flag is 0 during initialization and when no data has been returned; it is set to 1 after data is returned. This flag tracks the data retrieval status. When data is returned from the lower-level storage, the system sets this flag to 1 to facilitate subsequent processing of requests corresponding to this entry. Specifically, when a cache miss occurs for a request to a certain address, the system initiates a data retrieval request to the lower-level storage, records the relevant information of the request in an entry, and sets the isReturned flag of that entry to 0. After the lower-level storage returns the data, the system sets the isReturned flag of that entry to 1, at which point other requests in the waiting queue (if any) can begin processing the data.

[0088] The pending request FIFO (Fixed Request Forwarder) queue stores requests for the same memory address that are waiting for data to be returned. When multiple requests for the same address both result in cache misses, these requests are added to the pending request queue for the corresponding entry. Once the data is returned, the requests in the pending request queue are processed sequentially (e.g., first-in, first-out). For example, suppose two computing units simultaneously request data at the same address and both experience cache misses. The system allocates an entry for that address and adds both requests to the pending request queue for that entry. When the data is returned from the lower-level storage and loaded into the L1 cache, the system retrieves the requests from the pending request queue sequentially for processing, ensuring that all requests for that address receive the correct data. It should be understood that while cache miss requests for the same address are stored in the pending request queue and processed in a specific order, cache miss requests for different addresses are recorded in different entries. These requests can be handled flexibly without following a fixed processing order, thus improving processing efficiency.

[0089] Furthermore, considering the diverse characteristics of data in current AI models and high-performance computing, particularly the frequent occurrence of all-zero values, consider this: For instance, during AI model training, pruning techniques can optimize the model, reduce the number of parameters, and improve efficiency. Pruned models, such as BERT (Bidirectional Encoder Representations from Transformers) and ResNet (Residual Network), contain a large number of zero values ​​in their weight matrices. Specifically, the proportion of zero values ​​can reach 30% to 90%, depending on the model's sparsity. This weight sparsity provides advantages in storage and computation, but it also creates a need for processing all-zero data. Another example is in sequence data processing. To ensure consistent processing of sequences of different lengths, padding is often necessary. Taking the BERT model as an example, a certain number of zeros are padded to align the sequence length during input, with a padding ratio reaching 20% ​​to 30%. While these padded zeros do not contain actual information, they still need to be considered and processed during data processing.

[0090] Given these frequent all-zero data patterns, this embodiment of the invention implements a dedicated optimization mechanism for such data. Specifically, an all-zero flag (isAllZero) is introduced into the entries of the preset request management structure. This flag indicates whether the data at the address corresponding to the entry is entirely zero. By setting this flag, a zero-value detection mechanism can be implemented when data is returned, avoiding unnecessary subsequent processing. Specifically, when the system retrieves data from lower-level storage and returns it, it checks whether the data is entirely zero. For identified all-zero data, its metadata and isAllZero are maintained only in the entry of the preset request management structure; that is, the isAllZero value of the entry is set to 1, avoiding actual storage of all-zero data in the L1 cache and improving the space utilization of the L1 cache for non-zero data. When the same data address is accessed subsequently, the L1 cache reports a cache miss and then searches the preset request management structure. When the corresponding entry is found and isAllZero is 1 (i.e., true), the all-zero data can be returned directly without accessing the next level of storage.

[0091] Understandably, entries with all zeros (i.e., entries where isAllZero is true) can be persistently stored in the default request management structure, only being passively replaced when space is limited. By introducing isAllZero into the entries, the management of all-zero data is transferred from the L1 cache to the default request management structure, effectively reducing the pressure on the L1 cache and improving the efficiency of the overall storage system.

[0092] Based on any of the above embodiments, step 210 specifically includes:

[0093] Step 211: Based on the first address, determine the group index and entry tag;

[0094] Step 212: Based on the group index, determine the target group from multiple groups in the preset request management structure, and match the entry tag with the tag of each entry in the target group;

[0095] Step 213: If a match is successful, the matched entry is identified as the incomplete request entry.

[0096] Step 214: If the matching fails, it is determined that there is no incomplete request entry in the preset request management structure.

[0097] Specifically, when the caching system detects a cache miss in a data access request, it checks the preset request management structure for any incomplete request entries for that address, based on the first address carried in the request. Here, the system first extracts the group index (i.e., the set index) and the entry tag (i.e., the tag part) from the first address. Then, based on the group index, it locates the target group from multiple groups in the preset request management structure. Next, the system iterates through all entries in the target group, comparing the tag of each entry with the tag extracted from the first address. If the entry tag matches the tag of an entry in the target group, a match is successful; otherwise, it continues checking the next entry until all entries in the target group have been traversed. It should be understood that the target group is the group corresponding to the group index calculated based on the first address carried in the data access request; it is the group in the preset request management structure that may contain incomplete request entries related to the current request.

[0098] When the entry tag extracted from the first address is matched successfully with the tags of each entry in the target group, it means that there is already an incomplete request entry for the first address carried by the current data access request in the preset request management structure. In other words, other requests have previously experienced cache misses for the same address, and the relevant information of those requests has been recorded in the preset request management structure. In this case, this matched entry can be identified as the incomplete request entry corresponding to the current data access request, so that the current data access request can be added to the waiting queue corresponding to this entry. This allows the current data access request to share the processing progress of previous requests, avoiding repeated data access operations to lower-level storage.

[0099] If a match fails in step 212, it means that there is no incomplete request entry for the first address carried in the current data access request in the preset request management structure. In other words, no other requests for the same address have previously experienced cache misses, or previous requests for that address have been processed and removed from the preset request management structure. In this case, the system needs to allocate a new incomplete request entry for the current data access request so that the request can be processed subsequently.

[0100] Based on any of the above embodiments, step 220 specifically includes:

[0101] Step 221: Obtain the all-zero flag of the incomplete request entry.

[0102] Specifically, the purpose of this step is to determine whether the data at the address corresponding to an incomplete request entry is all zeros. In the default request management structure, each entry contains an all-zero flag (i.e., isAllZero), which is used to record the crucial information of whether the data at the corresponding address is all zeros.

[0103] Once an incomplete request entry corresponding to the first address carried in the data access request is found in the preset request management structure, the system will directly read the all-zero flag bit in that entry. Here, obtaining the all-zero flag bit is the key basis for subsequent processing. It determines whether the system needs to continue to send data requests to lower-level storage (such as L2 cache, memory, etc.), thereby optimizing the data processing flow and avoiding unnecessary resource waste.

[0104] Step 222: If the all-zero flag is true, return all-zero data directly to the data access request.

[0105] Specifically, when the all-zero flag obtained in step 221 is true (i.e., isAllZero=1), it indicates that the data at the address corresponding to the incomplete request entry is all 0. In this case, the system no longer stores the current data access request in the waiting queue of that entry, nor does it initiate a data request to a lower-level storage hierarchy. Instead, it directly returns all-zero data to the component (such as a computing unit) that initiated the current data access request. This is because since it has been determined that the data at that address is all 0, no matter how many times it is retrieved from the lower-level storage, the result will be all-zero data, so there is no need for additional data request operations. This processing method can significantly improve system efficiency, reduce unnecessary network transmission and storage access operations, and reduce system energy consumption and latency.

[0106] Step 223: If the all-zero flag is false, add the data access request to the waiting queue of the incomplete request entry.

[0107] Specifically, when the all-zero flag obtained in step 221 is false (i.e., isAllZero=0), it indicates that the data at the address corresponding to the incomplete request entry is not all zeros. In this case, the system can add the current data access request to the pending request FIFO queue of the incomplete request entry. After the system retrieves the data from the lower-level storage, it will load the data into the L1 cache and set the data return flag (isReturned) of the incomplete request entry to 1. Then, the system will process these requests sequentially according to the order in the pending queue and return the retrieved data to the corresponding request component.

[0108] In this embodiment of the invention, by adding requests to a waiting queue, the system can effectively manage multiple requests for the same address, ensuring data consistency and correctness, while avoiding repeated data requests to lower-level storage, thus improving the system's resource utilization.

[0109] Based on any of the above embodiments, after step 210, the method further includes:

[0110] Step 230: If there is no incomplete request entry in the preset request management structure, check whether there is a free entry in the target group of the preset request management structure. The target group is determined based on the first address carried by the data access request, and the free entry is determined based on the validity flag bit of each entry in the target group.

[0111] Specifically, when step 210 checks the preset request management structure and finds no incomplete request entry corresponding to the first address carried by the current data access request, it means there was no incomplete request for that address previously. In this case, it is necessary to further check if there are any free entries in the target group to store this new cache miss request. It should be understood that the target group is determined based on the first address carried by the data access request; the specific determination process can be referred to the above embodiment and will not be repeated here. Free entries are determined based on the validity flag of each entry in the target group. A false validity flag (i.e., valid=0) indicates that the entry is invalid, meaning it is in a free state and can be used by a new request.

[0112] Further, in step 230, checking whether there are any free entries in the target group of the preset request management structure includes:

[0113] Step 231: Traverse each entry in the target group;

[0114] Step 232: If the validity flag of the currently traversed entry is false, then the currently traversed entry is determined as the free entry.

[0115] Specifically, the system checks each entry in the target group sequentially, starting with the first entry and examining the status of its validity flag. During the iteration, if an entry's validity flag is found to be false (valid=0), it indicates that the entry is free and can be identified as such. This entry can then be used to record current cache misses. If no free entry is found after iterating through all entries in the target group, it is determined that there are currently no free entries in that target group.

[0116] Step 240: If there is a free entry in the target group, the free entry is assigned as an incomplete request entry corresponding to the first address.

[0117] Specifically, when it is determined that there is a free entry in the target group, this free entry can be assigned to the current data access request to record relevant information about the request.

[0118] Further, in step 240, allocating the free entry as an incomplete request entry corresponding to the first address includes:

[0119] Step 241: Store the entry tag carried in the first address into the tag flag bit of the free entry;

[0120] Step 242: Set the validity flag of the free entry to true, and set the data return flag and all-zero flag of the free entry to false;

[0121] Step 243: Add the data access request to the waiting queue of the idle entry, and initiate a data request to the lower-level storage.

[0122] Specifically, when assigning a found free entry to the current data access request, the entry tag extracted from the first address carried in the data access request can first be stored in the tag flag of the free entry. By storing the tag, the address corresponding to the entry can be uniquely identified, and the relevant entry can be quickly located based on this tag when processing the request or searching for data.

[0123] Then, the validity flag of the free entry is set to true (valid=1), indicating that the entry is currently occupied and an incomplete request is being processed. Simultaneously, the data return flag for this entry is set to false (isReturned=0), because data has not yet been retrieved from the lower-level storage and has not been returned. Furthermore, the all-zero flag for this entry needs to be set to false (isAllZero=0), because it is currently unknown whether the data corresponding to this address is all zeros; this needs to be determined after data is retrieved from the lower-level storage. These flag settings accurately reflect the current status of the entry, providing a basis for subsequent request processing and status determination.

[0124] After setting the relevant flags, the current data access request can be added to the pending request FIFO of the idle entry. Since the request resulted in a cache miss and was stored in a newly allocated entry, this means there were no other pending requests for this address before. Therefore, data needs to be retrieved from lower-level storage (such as L2 cache, memory, etc.). So the system will send a data request to the lower-level storage and wait for the data to be returned before processing the pending requests in this entry.

[0125] Based on any of the above embodiments, after step 230, the method further includes:

[0126] If no free entry exists in the target group, check if there is an all-zero entry in the target group of the preset request management structure. The all-zero entry is determined based on the all-zero flag bit of each entry in the target group.

[0127] If it exists, the all-zero entry is replaced so that the all-zero entry is assigned as the incomplete request entry corresponding to the first address;

[0128] If it does not exist, the data access request is determined to be blocked, and the process waits for the entry in the target group to be released.

[0129] Specifically, in step 230 above, when the check finds that there are no free entries in the target group (i.e., all entries in the target group have a valid value of true, and there are no available entries to record new cache miss requests), the system will check each entry in the target group again to see if the all-zero flag (isAllZero) of each entry is true. If the isAllZero flag of an entry is true, then that entry is identified as an all-zero entry. It should be understood that in this embodiment of the invention, when there are no free entries in the target group, an attempt is made to find an all-zero entry to replace it in order to store the new cache miss request.

[0130] When an all-zero entry is found in the target group, the relevant information in the all-zero entry can be replaced to assign the entry as an incomplete request entry corresponding to the current data access request. Specifically, the entry tag extracted from the first address carried in the data access request can be stored in the tag flag of the all-zero entry, thereby replacing the address information corresponding to the original all-zero entry with the new address information. Then, the validity flag of the all-zero entry is set to true (valid=1), the data return flag is set to false (isReturned=0), and the all-zero flag is set to false (isAllZero=0). Finally, the current data access request is added to the pending request FIFO of the entry, and a data request is initiated to the lower-level storage. It should be understood that, by replacing all-zero entries, this embodiment of the invention can still handle new cache miss requests even when there are no free entries in the target group, improving the system's resource utilization and processing capacity.

[0131] When the check finds no all-zero entries in the target group, it means there are neither free entries nor all-zero entries available for replacement. The system cannot immediately process the data access request and therefore classifies it as blocked. The system waits for entries in the target group to be released. For example, after an entry completes its request, its validity flag is set to false, indicating that the entry becomes free. Once an entry is released, the system re-attempts to process the data access request, returning to step 230 to re-check the target group. It should be understood that, in situations of insufficient resources, blocking requests and waiting for resource release ensures system stability and data consistency, avoiding erroneous processing due to resource shortages.

[0132] Understandably, all-zero entries typically employ a persistent storage strategy in the default request management structure, passively replacing them only when space is limited. Here, persistent storage means that all-zero entries are retained in the default request management structure for an extended period and are not easily replaced. The purpose of this strategy is to enable rapid request response in certain specific data scenarios (such as scenarios where all-zero data frequently occurs), reducing the overhead caused by frequent entry replacement. Replacement of all-zero entries is only considered when the target group's space is limited, i.e., there are no free entries and no other replaceable entries. This passive replacement method ensures that all-zero entries are retained in most cases, improving the system's efficiency when processing all-zero data.

[0133] Based on any of the above embodiments, the method further includes:

[0134] Step 250: After receiving the target data returned by the lower-level storage, based on the second address carried by the target data, search for a target request entry that matches the second address in the preset request management structure.

[0135] Specifically, for each incomplete request recorded in the preset request management structure, when the caching system receives the target data for a certain request returned by the lower-level storage, it can search for the target request entry that matches the second address in the preset request management structure based on the second address carried by the target data, thereby triggering the execution of all requests in the waiting queue of that entry.

[0136] Here, target data refers to the data obtained from lower-level storage (such as L2 cache, memory, etc.) that corresponds to a specific data access request. For example, if a computing unit requests access to data at a certain address, but that data is not in the L1 cache, the system will send a request to the L2 cache, and the data returned by the L2 cache is the target data.

[0137] The target data returned by the lower-level storage is typically associated with a specific address, which is called the second address. In a caching system, in order to correctly store the target data in the cache (such as the L1 cache) and process related requests, it is necessary to know the address corresponding to the target data. The target request entry refers to the entry in the predefined request management structure that matches the second address carried by the target data.

[0138] Specifically, when the lower-level storage returns the target data to the caching system, the caching system extracts the set index and entry tag from the second address carried in the target data. Then, based on the set index, it locates the corresponding group in the preset request management structure. Next, it iterates through each entry in that group, comparing the tag of each entry with the tag extracted from the second address. If the tag of an entry matches the tag in the second address, then that entry is the target request entry.

[0139] Step 260: Write the target data into the cache and set the data return flag of the target request entry to true to trigger the execution of all requests in the waiting queue of the target request entry.

[0140] Specifically, after locating the target request entry, the target data is first loaded into the cache (such as the L1 cache) so that subsequent accesses can hit the cache. Then, the data return flag of the target request entry is set to true (i.e., isReturned=1), indicating that the data corresponding to this entry has been returned from the lower-level storage and written to the cache. By setting the data return flag to true, the system can be triggered to process all requests in the waiting queue of the target request entry.

[0141] Based on any of the above embodiments, step 260, writing the target data into the cache, includes:

[0142] Step 261: If the target data is all zeros, set the all-zero flag of the target request entry to true;

[0143] Step 262: If the target data is not all zeros, write the target data into the cache.

[0144] Specifically, considering that the target data returned by the lower-level storage may be all zeros, it is necessary to detect and identify all-zero data when returning the data. Here, methods such as byte-by-byte (or bit-by-bit) comparison or hardware instructions can be used to detect and identify all-zero data. For example, each byte (or bit) of the target data can be traversed and compared with 0. If the value of all bytes (or bits) is 0, the target data is determined to be all-zero; otherwise, it is determined to be non-all-zero data. Another example is to use hardware instructions to perform a bitwise OR operation on the target data, and make a quick preliminary judgment based on the operation result. If the hardware instructions cannot directly produce a result, then a byte-by-byte comparison method can be used for a more precise judgment.

[0145] When the target data is all zeros, simply set the all-zero flag of the corresponding target request entry to true (i.e., isAllZero=1). There's no need to write this all-zero data into the L1 cache, thus avoiding actual storage of all-zero data in the L1 cache and improving its utilization efficiency for non-zero data. Furthermore, by setting the all-zero flag of the target request entry to true, subsequent requests accessing the address of that entry can quickly return all-zero data based on the flag, avoiding unnecessary cache accesses and lower-level storage requests.

[0146] When the target data is not all zeros, it needs to be loaded into the L1 cache so that subsequent accesses can hit the cache and improve system performance.

[0147] Based on any of the above embodiments, step 260, which involves executing all requests in the waiting queue of the target request entry, includes:

[0148] Step 263: Each cycle, take one request from the waiting queue of the target request entry and return the target data to the request, until the waiting queue of the target request entry is empty, and then release the target request entry.

[0149] Specifically, when executing all requests in the target request entry waiting queue, the system can retrieve one request from the waiting queue of the target request entry every clock cycle and return the target data to the component that initiated the request (such as the computing unit). When all requests in the waiting queue have been processed, i.e., when the waiting queue is empty, the target request entry can be released so that the entry can be reused.

[0150] Understandably, in practical applications, the default request management structure typically involves multiple entries whose cache misses are processed in parallel, with each entry having its own waiting queue. At each clock cycle, the system needs to determine which entries' waiting queues are "serviceable" (i.e., entries with `isReturned` equal to 1 and a non-empty `pending request FIFO` are serviceable). Selecting which entry to service from these serviceable entries can be achieved through a priority-based round-robin scheduling method. Specifically, at each clock cycle, all entries are polled, and the first entry with `isReturned` equal to 1 and a non-empty `pending request FIFO` is found. The head request in that entry's waiting queue is then processed. In the next clock cycle, polling begins from the next entry, ensuring fairness.

[0151] Further, in step 263, releasing the target request entry includes:

[0152] Clear the flag of the target request entry, set the validity flag and the data return flag of the target request entry to false, and clear the waiting queue of the target request entry.

[0153] Specifically, after processing all requests in the waiting queue of a target request entry, the target request entry can be released. Specifically, first, the tag stored in the target request entry can be cleared, indicating that the entry is no longer associated with a specific address; then, the validity flag and data return flag of the target request entry are set to false (i.e., valid=0, isReturned=0), indicating that the entry is invalid and no data is returned. Finally, the waiting queue of the target request entry is emptied to ensure that no residual request information remains.

[0154] In this embodiment of the invention, the target request entry is restored to its initial idle state so that it can be subsequently reassigned to new cache miss requests.

[0155] Based on any of the above embodiments Figure 4 This is a flowchart illustrating the parallelized cache miss request handling method based on a group-associative cache structure provided by the present invention, as shown below. Figure 4 As shown, this embodiment of the invention redesigns the traditional FIFO queue structure into a multi-way set-associative structure (such as an 8-way set-associative structure), achieving efficient group management of cache miss requests. This improved multi-way set-associative structure (i.e., the preset request management structure) is named mCache, which allocates cache miss requests to different groups based on their addresses, significantly optimizing query efficiency when returning data. The following are the specific steps of this method:

[0156] S1, Cache miss detected:

[0157] When the computing unit accesses address A through a data load instruction, it first searches the L1 cache. If the data is not in the L1 cache, an L1 cache miss will be detected.

[0158] S2, Assigning Entries:

[0159] First, check if there is already an incomplete request entry for address A in mCache. If it exists, further determine if the entry is an all-zero entry. If the isAllZero of the entry is 1, then return all-zero data directly to the current request (without waiting for data to be returned from the lower-level storage); otherwise, add the current request to the waiting queue (pendingrequest FIFO) of the entry.

[0160] If no incomplete request entry for address A exists in mCache, then the system checks if there is a free entry (i.e., an entry with a valid value of 0). If a free entry exists, it is allocated to the current request to record relevant information. Specifically, the tag portion of address A is stored in the tag of the free entry, the entry's validity flag is set to 1, and its all-zero flag and data return flag are set to 0 (i.e., valid=1, isReturned=0, isAllZero=0), and the current request is added to the entry's waiting queue. Subsequently, a data request is initiated to lower-level storage (such as L2 or memory).

[0161] If no free entry is found, the system checks if an all-zero entry exists in the mCache. If it does, the all-zero entry is replaced, meaning it's assigned to the current request to record relevant information. Specifically, the tag of the all-zero entry is replaced with the tag portion from address A, the entry's validity flag is set to 1, and the all-zero flag and data return flag are set to 0 (i.e., valid=1, isReturned=0, isAllZero=0). The current request is then added to the entry's waiting queue. Subsequently, a data request is initiated to lower-level storage (such as L2 or memory). If no all-zero entry exists, the current request stalls (blocks), waiting for an entry to be released before retrying.

[0162] S3, Data returned:

[0163] After the lower-level storage returns data, the data carries the corresponding address information. Based on this address information, a matching entry is searched in mCache (using tag matching). Once a matching entry is found, the data return flag (i.e., isReturned) for that entry is set to 1, and the data is written to the L1 cache. Here, if the lower-level storage returns all-zero data, the all-zero flag (i.e., isAllZero) for that entry is set to 1, and there is no need to write the data to the L1 cache. Subsequently, each cycle, one request is taken from the pendingrequest FIFO queue, and data is returned until the queue is empty.

[0164] S4, Release entry:

[0165] After processing all requests in the waiting queue for a specific entry, the contents of that entry (including tag information) can be cleared from mCache. Simultaneously, the entry's validity flag (i.e., valid) is set to 0, the data return flag (i.e., isReturned) is set to 0, and the pending request FIFO queue is cleared. This entry then becomes idle and available for subsequent requests. It should be understood that, as... Figure 4 As shown, for all-zero entries, after returning all-zero data to the current request, it is usually not necessary to release the entry, because all-zero entries are persistently stored in mCache. However, if there are multiple requests in the waiting queue of the entry, the waiting queue of the entry can be cleared after these requests are processed.

[0166] The method provided in this invention, by employing a group-associative structure instead of the traditional FIFO management structure, achieves parallel lookup of cache miss requests, thereby improving the processing efficiency of cache miss requests. Furthermore, by introducing `isAllZero` into the entries, all-zero data can be identified and specially processed, avoiding the storage of all-zero data in the L1 cache and improving the utilization of L1 cache space.

[0167] The request management device provided by the present invention will be described below. The request management device described below can be referred to in correspondence with the request management method described above.

[0168] Based on any of the above embodiments Figure 5 This is a schematic diagram of the request management device provided by the present invention, as shown below. Figure 5 As shown, the device includes:

[0169] The checking unit 510 is used to check whether there is an incomplete request entry corresponding to the first address in the preset request management structure when a cache miss is detected in a data access request, based on the first address carried by the data access request. The preset request management structure is a pre-built multi-way group-associative cache structure for grouping and managing cache miss requests.

[0170] The addition unit 520 is used to add the data access request to the waiting queue of the incomplete request entry when there is an incomplete request entry in the preset request management structure.

[0171] The apparatus provided in this invention, when a cache miss is detected in a data access request, checks whether a corresponding incomplete request entry exists in a preset request management structure based on the first address carried in the request. If such an entry exists, the request can be added to the waiting queue of that entry. This means that when a request with the same address is detected, subsequent requests can be directly added to the waiting queue of existing requests, effectively merging cache miss requests with the same address. This eliminates the need to send multiple requests to lower-level storage for the same address, thus avoiding the overhead of repeatedly accessing lower-level storage and significantly reducing system bandwidth consumption and access latency. Furthermore, the preset request management structure is a pre-built multi-way group-associative cache structure. By using this group-associative cache structure instead of the traditional FIFO structure, group management and parallel lookup of cache miss requests can be achieved. This effectively avoids problems such as queue head blocking and inability to process out-of-order requests caused by sequential processing in the traditional FIFO structure, fully utilizing the parallel characteristics of the memory system and significantly improving the processing efficiency of cache miss requests.

[0172] Based on any of the above embodiments, the joining unit 520 is specifically used for:

[0173] Obtain the all-zero flag of the incomplete request entry;

[0174] If the all-zero flag is true, return all-zero data directly to the data access request;

[0175] If the all-zero flag is false, the data access request is added to the waiting queue of the incomplete request entry.

[0176] Based on any of the above embodiments, the preset request management structure includes multiple groups, each group includes multiple entries, and the inspection unit 510 is specifically used for:

[0177] Based on the first address, determine the group index and entry tag;

[0178] Based on the group index, a target group is determined from multiple groups in the preset request management structure, and the entry tag is matched with the tag of each entry in the target group;

[0179] If a match is found, the matched entry will be identified as the incomplete request entry.

[0180] If the matching fails, it is determined that there is no incomplete request entry in the preset request management structure.

[0181] Based on any of the above embodiments, the apparatus further includes an entry allocation unit, the entry allocation unit being used for:

[0182] If there is no incomplete request entry in the preset request management structure, check if there is a free entry in the target group of the preset request management structure. The target group is determined based on the first address carried by the data access request, and the free entry is determined based on the validity flag bit of each entry in the target group.

[0183] If a free entry exists in the target group, the free entry will be assigned as an incomplete request entry corresponding to the first address.

[0184] Based on any of the above embodiments, the entry allocation unit is specifically used for:

[0185] Store the entry tag carried in the first address into the tag flag bit of the free entry;

[0186] Set the validity flag of the free entry to true, and set the data return flag and all-zero flag of the free entry to false;

[0187] The data access request is added to the waiting queue of the idle entry, and a data request is initiated to the lower-level storage.

[0188] Based on any of the above embodiments, the device further includes an entry replacement unit, the entry replacement unit being used for:

[0189] If no free entry exists in the target group, check if there is an all-zero entry in the target group of the preset request management structure. The all-zero entry is determined based on the all-zero flag bit of each entry in the target group.

[0190] If it exists, the all-zero entry is replaced so that the all-zero entry is assigned as the incomplete request entry corresponding to the first address;

[0191] If it does not exist, the data access request is determined to be blocked, and the process waits for the entry in the target group to be released.

[0192] Based on any of the above embodiments, the apparatus further includes a request execution unit, the request execution unit being configured to:

[0193] After receiving the target data returned by the lower-level storage, based on the second address carried by the target data, the target request entry matching the second address is searched in the preset request management structure;

[0194] The target data is written to the cache, and the data return flag of the target request entry is set to true to trigger the execution of all requests in the waiting queue of the target request entry.

[0195] Based on any of the above embodiments, the request execution unit is specifically used for:

[0196] If the target data is all zeros, set the all-zero flag of the target request entry to true;

[0197] If the target data is not all zeros, the target data is written to the cache.

[0198] Based on any of the above embodiments, the request execution unit is specifically used for:

[0199] Each cycle, a request is taken from the waiting queue of the target request entry, and the target data is returned to the request, until the waiting queue of the target request entry is empty, at which point the target request entry is released.

[0200] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a request management method, which includes: when a cache miss is detected for a data access request, checking whether there is an incomplete request entry corresponding to the first address in a preset request management structure based on the first address carried by the data access request, wherein the preset request management structure is a pre-constructed multi-way set-associative cache structure for grouping and managing cache miss requests; and if the incomplete request entry exists in the preset request management structure, adding the data access request to the waiting queue of the incomplete request entry.

[0201] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the request management method provided by the above methods. The method includes: when a cache miss is detected in a data access request, checking whether there is an incomplete request entry corresponding to the first address in a preset request management structure based on the first address carried by the data access request, the preset request management structure being a pre-constructed multi-way set-associative cache structure for grouping and managing cache miss requests; and when the incomplete request entry exists in the preset request management structure, adding the data access request to the waiting queue of the incomplete request entry.

[0203] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the request management method provided by the above methods. The method includes: when a cache miss is detected in a data access request, checking whether there is an incomplete request entry corresponding to the first address in a preset request management structure based on a first address carried by the data access request, wherein the preset request management structure is a pre-constructed multi-way set-associative cache structure for grouping and managing cache miss requests; and if the incomplete request entry exists in the preset request management structure, adding the data access request to the waiting queue of the incomplete request entry.

[0204] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0205] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A request management method, characterized in that, include: When a cache miss is detected in a data access request, based on the first address carried in the data access request, it is checked whether there is an incomplete request entry corresponding to the first address in the preset request management structure. The preset request management structure is a pre-built multi-way group-associative cache structure used to merge and group cache miss requests. If an incomplete request entry exists in the preset request management structure, obtain the all-zero flag bit of the incomplete request entry; When the all-zero flag is true, it indicates that the data at the address corresponding to the incomplete request entry is all zero, and all-zero data is directly returned to the data access request to avoid actually storing all-zero data in the cache; If the all-zero flag is false, the data access request is added to the waiting queue of the incomplete request entries to merge multiple cache miss requests for the first address.

2. The request management method according to claim 1, characterized in that, The preset request management structure includes multiple groups, and each group includes multiple entries. The step of checking whether there is an incomplete request entry corresponding to the first address in the preset request management structure based on the first address carried in the data access request includes: Based on the first address, determine the group index and entry tag; Based on the group index, a target group is determined from multiple groups in the preset request management structure, and the entry tag is matched with the tag of each entry in the target group; If a match is found, the matched entry will be identified as the incomplete request entry. If the matching fails, it is determined that there is no incomplete request entry in the preset request management structure.

3. The request management method according to claim 1, characterized in that, Also includes: If there is no incomplete request entry in the preset request management structure, check if there is a free entry in the target group of the preset request management structure. The target group is determined based on the first address carried by the data access request, and the free entry is determined based on the validity flag bit of each entry in the target group. If a free entry exists in the target group, the free entry will be assigned as an incomplete request entry corresponding to the first address.

4. The request management method according to claim 3, characterized in that, The step of assigning the free entry as the incomplete request entry corresponding to the first address includes: Store the entry tag carried in the first address into the tag flag bit of the free entry; Set the validity flag of the free entry to true, and set the data return flag and all-zero flag of the free entry to false; The data access request is added to the waiting queue of the idle entry, and a data request is initiated to the lower-level storage.

5. The request management method according to claim 3, characterized in that, Also includes: If no free entry exists in the target group, check if there is an all-zero entry in the target group of the preset request management structure. The all-zero entry is determined based on the all-zero flag bit of each entry in the target group. If it exists, the all-zero entry is replaced so that the all-zero entry is assigned as the incomplete request entry corresponding to the first address; If it does not exist, the data access request is determined to be blocked, and the process waits for the entry in the target group to be released.

6. The request management method according to any one of claims 1 to 5, characterized in that, Also includes: After receiving the target data returned by the lower-level storage, based on the second address carried by the target data, the target request entry matching the second address is searched in the preset request management structure; The target data is written to the cache, and the data return flag of the target request entry is set to true to trigger the execution of all requests in the waiting queue of the target request entry.

7. The request management method according to claim 6, characterized in that, The step of writing the target data into the cache includes: If the target data is all zeros, set the all-zero flag of the target request entry to true; If the target data is not all zeros, the target data is written to the cache.

8. The request management method according to claim 6, characterized in that, Executing all requests in the waiting queue of the target request entry includes: Each cycle, a request is taken from the waiting queue of the target request entry, and the target data is returned to the request, until the waiting queue of the target request entry is empty, at which point the target request entry is released.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the request management method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the request management method as described in any one of claims 1 to 8.

Images