Cache device and operating method, electronic device, and artificial intelligence processor

Abstract

This disclosure provides a caching device and operating method, an electronic device, and an artificial intelligence processor. The caching device includes a receiving module, a scheduling module, and an execution module. The receiving module is configured to receive multiple requests; the scheduling module includes multiple waiting queues and is configured to store the multiple requests into the corresponding waiting queues based on their request types. The multiple waiting queues include a read / write request queue and a computation request queue; the execution module includes multiple execution pipelines, each coupled to a corresponding waiting queue in the multiple waiting queues. The execution pipelines also include read / write pipelines and computation pipelines. The read / write pipelines are configured to execute read / write requests emitted from the read / write request queue, and the computation pipelines are configured to execute computation requests emitted from the computation request queue. This caching device can improve the computational efficiency of general-purpose graphics processing units.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of integrated circuits, and more specifically to a cache device and operating method, an electronic device, and an artificial intelligence processor. Background Technology

[0002] In general-purpose graphics processors, the lowest level cache (LLC) is the lowest-level on-chip global cache. To alleviate the shortage of general-purpose registers and shared memory, data reduction and atomic operations can be offloaded to the LLC to perform in-memory / near-memory computations, reducing data transfer overhead. However, integrating computational functions into the LLC deepens the pipeline and complicates the logic, causing numerous problems. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a caching device, comprising: a receiving module, a scheduling module, and an execution module. The receiving module is configured to receive multiple requests; the scheduling module includes multiple waiting queues and is configured to store the multiple requests into the multiple waiting queues respectively based on the request type of the multiple requests, wherein the multiple waiting queues include a read / write request queue and a computation request queue; and the execution module includes multiple execution pipelines, wherein the multiple execution pipelines are respectively coupled to corresponding waiting queues in the multiple waiting queues, and the multiple execution pipelines include a read / write pipeline and a computation pipeline; wherein the read / write pipeline is configured to execute read / write requests emitted from the read / write request queue, and the computation pipeline is configured to execute computation requests emitted from the computation request queue.

[0004] For example, in the cache device provided in at least one embodiment of this disclosure, the read / write request queue includes a main read / write request queue and a secondary read / write request queue; the read / write pipeline includes a main read / write request pipeline and a secondary read / write request pipeline. The scheduling module is further configured to: in response to the main read / write request queue transmitting a first read / write request, allocate a second read / write request with the same access address as the first read / write request to the secondary read / write request queue; or, in response to the main read / write request queue transmitting a first read / write request, the secondary read / write request queue including a third read / write request, and the third read / write request not being transmitted in the current clock cycle, not allocate the request to the secondary read / write request queue, wherein the access address corresponding to the third read / write request is different from that of the first read / write request.

[0005] For example, in a cache device provided in at least one embodiment of this disclosure, the scheduling module is further configured to, after the read / write request main queue sends out the first read / write request, send out the second read / write request in response to the second read / write request passing a resource check.

[0006] For example, in a cache device provided in at least one embodiment of this disclosure, the resource check includes checking whether the second read / write request conflicts with other requests issued during the current clock cycle for access to a storage unit.

[0007] For example, in a cache device provided in at least one embodiment of this disclosure, the read / write request sub-queue is configured to send the second read / write request within at least one clock cycle after the read / write request main queue sends the first read / write request.

[0008] For example, in at least one embodiment of the caching device provided in this disclosure, the caching device further includes a storage module, wherein the execution module includes a write data cache and a credit cache. The read / write request mainline is configured to, in response to the first read / write request being a read request, access the storage module to read target data and provide the target data to the credit cache; or, in response to the first read / write request being a write request, retrieve source data from the write data cache and write it to the storage module.

[0009] For example, in a cache device provided in at least one embodiment of this disclosure, the storage module includes a first storage unit or a second storage unit; the read / write request mainline is further configured to access the first storage unit or the second storage unit according to the parity of the cache line number to execute the read request or the write request.

[0010] For example, in the caching device provided in at least one embodiment of this disclosure, the execution module further includes a forwarding register; the main read / write request pipeline is further configured to, in response to the first read / write request being a read request or a write request, write the target data or the source data into the forwarding register. The secondary read / write request pipeline is further configured to: in response to the second read / write request being a read request, read the first data from the forwarding register; or, in response to both the second read / write request being a write request and the first read / write request being a write request, obtain the source data from the forwarding register, merge it with the write data of the second read / write request, and write the merged data into the storage module.

[0011] For example, in the cache device provided in at least one embodiment of this disclosure, the execution module further includes a plurality of computing units and an arithmetic logic unit cache; the computing pipeline is further configured to retrieve first source data from the arithmetic logic unit cache, and enter the corresponding computing unit among the plurality of computing units to perform the corresponding computing operation according to the operation type corresponding to the computing request, and write the corresponding computing result into the storage module.

[0012] For example, in a cache device provided in at least one embodiment of this disclosure, the computing pipeline is further configured to read second source data from the storage module and perform computing operations on the second source data and the first source data.

[0013] For example, in the cache device provided in at least one embodiment of this disclosure, the scheduling module is further configured such that the launch priority of the main queue of read / write requests is higher than that of the computation request queue, and the launch priority of the computation request queue is higher than that of the secondary queue of read / write requests.

[0014] For example, in at least one embodiment of the caching device provided in this disclosure, the caching device further includes: a storage module and a hit detection module. The storage module includes multiple cache lines; the hit detection module is configured to: in response to a target read / write request in the multiple requests hitting a target cache line in the storage module, adjust the status information of the target cache line according to the request type of the multiple requests, and provide the target read / write request to the scheduling module; or, in response to a target read / write request in the multiple requests not hitting a target cache line in the storage module, provide the target read / write request to the scheduling module. The scheduling module is further configured to: in response to a target read / write request not hitting a target cache line in the storage module, send a read request to memory to obtain the target data corresponding to the target read / write request, and write it to the target cache line.

[0015] For example, in a caching device provided in at least one embodiment of this disclosure, the scheduling module further includes a request cache configured to temporarily store the at least one request that has undergone a hit detection.

[0016] For example, in a caching device provided in at least one embodiment of this disclosure, the scheduling module is further configured to write the data in the target cache line into the memory before operating the target cache line according to the target read / write request, in response to the data in the target cache line being modified and not yet written to memory.

[0017] For example, in at least one embodiment of the caching device provided in this disclosure, the caching device further includes a request return module. The request return module is configured to provide a corresponding request response to the source of the target request via an on-chip bus in response to the completion of execution of a target request among the plurality of requests.

[0018] At least one embodiment of this disclosure provides a cache operation method, which is applied to a cache device provided in any embodiment of this disclosure. The cache device includes multiple execution pipelines and multiple wait queues. The multiple execution pipelines are respectively coupled to corresponding wait queues in the multiple wait queues, and the multiple execution pipelines include read-write pipelines and computation pipelines. The cache operation method includes: receiving multiple requests; storing the multiple requests into the multiple wait queues according to their request types, wherein the multiple wait queues include read-write request queues and computation request queues; executing read-write requests emitted from the read-write request queues through the read-write pipelines, and executing computation requests emitted from the computation request queues through the computation pipelines.

[0019] At least one embodiment of this disclosure provides an electronic device, which includes a cache device provided in any embodiment of this disclosure.

[0020] At least one embodiment of this disclosure provides an electronic device, the electronic device including: at least one processor and at least one memory, wherein the at least one memory stores at least one computer program, and when the at least one computer program is executed by the at least one processor, it implements the cache operation method provided in any embodiment of this disclosure.

[0021] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer-readable instructions non-transitory. When the computer-readable instructions are executed by a computer, the cache operation method provided in any embodiment of this disclosure is implemented.

[0022] At least one embodiment of this disclosure provides an artificial intelligence processor, which includes a cache device provided in any embodiment of this disclosure.

[0023] In at least one embodiment of this disclosure, by dividing the waiting queue in the scheduling module into a read / write request queue and a computation request queue according to the request type, and setting up independent execution pipelines coupled to each waiting queue in the execution module, decoupling and parallel processing of read / write requests and computation requests are achieved. This architecture allows read / write operations and computation operations to be scheduled and executed independently, effectively solving the performance degradation problem caused by mixed requests blocking each other in the same pipeline, and effectively improving the throughput of the cache device. In at least one embodiment, by setting up a dedicated computation pipeline for computation requests, tasks such as atomic operations and reduction calculations can be devolved to the cache level for execution without affecting the original read / write bandwidth, thereby alleviating the resource pressure on general-purpose registers and shared memory inside the computing core and improving the overall computing efficiency of the general-purpose graphics processor. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0025] Figure 1 A schematic diagram of the structure of a general-purpose graphics processing unit (GPGPU) is shown.

[0026] Figure 2 A schematic block diagram of a cache device provided in at least one embodiment of the present disclosure is shown.

[0027] Figure 3 A schematic block diagram of another cache device provided in at least one embodiment of the present disclosure is shown.

[0028] Figure 4 A schematic diagram illustrating the application of a cache device provided in at least one embodiment of the present disclosure is shown.

[0029] Figure 5 A schematic diagram illustrating the application of another cache device provided in at least one embodiment of the present disclosure is shown.

[0030] Figure 6 A schematic diagram illustrating the application of another cache device provided in at least one embodiment of the present disclosure is shown.

[0031] Figure 7 A schematic flowchart of a cache operation method provided in at least one embodiment of the present disclosure is shown.

[0032] Figure 8 A schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure is shown.

[0033] Figure 9 A schematic block diagram of another electronic device provided in at least one embodiment of the present disclosure is shown.

[0034] Figure 10 A schematic block diagram of a non-transitory computer-readable storage medium provided in at least one embodiment of the present disclosure is shown.

[0035] Figure 11 A schematic block diagram of an artificial intelligence processor provided in at least one embodiment of the present disclosure is shown. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0037] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0038] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numerals in each drawing.

[0039] Figure 1 A schematic diagram of the structure of a general-purpose graphics processor (GPGPU) is shown.

[0040] In parallel computing, computational tasks are typically executed using multiple threads. For example... Figure 1As shown, before these threads execute in the General-Purpose Graphics Processing Unit (GPGPU) (or parallel computing processor), they are divided into multiple thread blocks in the thread block scheduling module. Then, the thread block distribution module distributes these thread blocks to various computing units (CUs) (e.g., streaming multiprocessors (SMs)). All threads within a thread block must be assigned to the same computing unit for execution. Simultaneously, thread blocks are further divided into minimum execution thread warps (or simply thread warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute within the same computing unit or in different computing units.

[0041] Within each computing unit, the thread bundle scheduling / distribution module schedules and allocates thread bundles so that multiple computing cores (e.g., stream processors (SPs)) of that unit can run the thread bundles. Each computing core includes an arithmetic logic unit (ALU), a floating-point unit, etc. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can execute concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Instruction fetching, decoding, and issuing are all completed within the thread bundle scheduling / distribution module. Memory execution instructions are issued to shared caches (e.g., shared L1 caches) within the computing unit or further issued to a unified cache for read / write operations, etc.

[0042] When a thread executes a computation instruction, the required source data (from a general-purpose register) may come from a previous memory read instruction. In this case, a wait instruction is needed to ensure that the data read back by the previous memory read instruction is ready. Similar synchronization relationships exist between multiple thread bundles within the same thread block. For example, a thread block may consist of two thread bundles, the first and second, both of which require reading data from memory region A for computation. To save read time and bandwidth, a common optimization is for the first thread bundle to read half of the data in memory region A, and the second thread bundle to read the other half. However, for any given thread bundle, all the data in memory region A is required when executing a computation instruction. Therefore, before executing a computation instruction, the thread bundle needs to wait for the memory read instructions from both the first and second thread bundles to have read back the data. A barrier instruction is needed to prevent the thread bundle from continuing to execute instructions until the memory read instructions from the first and second thread bundles have finished, thus obtaining all the data in memory region A.

[0043] In the field of Graphics Processing Unit (GPU) design, especially in general-purpose GPUs (GPGPUs), the Last Level Cache (LLC) is the lowest level of the global cache (on-chip cache) hierarchy, providing high-bandwidth data access to the computing cores. With the widespread application of GPGPUs in general-purpose computing, artificial intelligence, and high-performance computing, the general-purpose registers and shared memory resources within the computing cores are becoming increasingly scarce, becoming a key bottleneck restricting performance. To alleviate this resource pressure, for example, some computational tasks (e.g., data reduction and atomic operations) can be offloaded to the LLC for execution. This "in-memory computation" or "near-memory computation" approach can effectively reduce the overhead of data movement between the computing cores and the cache, improving overall execution efficiency.

[0044] For example, to optimize read and write requests, the LLC can employ a single or shallow pipeline structure, resulting in a simpler request scheduling strategy. For instance, the LLC's processing units are primarily configured to read and write data based on the address of memory access requests, leading to relatively simple scheduling logic and lower hardware overhead.

[0045] However, as LLC begins to undertake more complex functions such as atomic operations and reduction calculations, its internal pipeline depth increases significantly, and its processing logic becomes more complex. The inventors of this disclosure note that the aforementioned LLC architecture also suffers from at least one of the following technical problems.

[0046] For example, when LLC integrates computing functions, the increased number of pipeline stages leads to increased request scheduling complexity. Single-issue or simple scheduling mechanisms cannot fully utilize pipeline resources, which can easily cause pipeline stalls and create performance bottlenecks.

[0047] For example, in typical GPGPU workloads, memory access requests from multiple warps often exhibit strong address dependencies, such as multiple requests simultaneously accessing the same cache line for accumulation or reduction operations. However, the aforementioned LLC design lacks scheduling optimizations for this address dependency, failing to leverage the dependencies between requests to reduce redundant cache read / write operations, resulting in wasted memory access bandwidth and increased power consumption.

[0048] For example, the LLC's request execution unit uses a single data path, making it difficult to efficiently handle mixed accesses of read / write requests and compute requests within the same cycle, especially when there are resource conflicts (such as cache group memory conflicts), resulting in low hardware utilization.

[0049] Therefore, how to design a hardware architecture that can efficiently schedule mixed requests with address dependencies, make full use of pipeline resources, and reduce redundant operations, while integrating computing functions into LLC, has become an urgent technical problem to be solved.

[0050] Based on at least one of the aforementioned technical problems, at least one embodiment of this disclosure provides a caching device, a caching operation method, an electronic device, and an artificial intelligence processor. The caching device includes a receiving module, a scheduling module, and an execution module. The receiving module is configured to receive multiple requests; the scheduling module includes multiple waiting queues and is configured to store the multiple requests into the multiple waiting queues respectively based on the request types of the multiple requests, wherein the multiple waiting queues include a read / write request queue and a computation request queue; the execution module includes multiple execution pipelines, wherein the multiple execution pipelines are respectively coupled to corresponding waiting queues in the multiple waiting queues, and the multiple execution pipelines also include read / write pipelines and computation pipelines, wherein the read / write pipelines are configured to execute read / write requests emitted from the read / write request queue, and the computation pipelines are configured to execute computation requests emitted from the computation request queue.

[0051] This high-speed cache device allows read / write operations and computation operations to be scheduled and executed independently, effectively solving the performance degradation problem caused by mixed requests blocking each other in the same pipeline, and effectively improving the throughput of the cache device. In at least one embodiment, by setting up a dedicated computation pipeline for computation requests, tasks such as atomic operations and reduction computations can be devolved to the cache level for execution without affecting the original read / write bandwidth, thereby alleviating the resource pressure on general-purpose registers and shared memory inside the computing core and improving the overall computing efficiency of, for example, GPGPUs.

[0052] Figure 2 A schematic block diagram of a cache device provided in at least one embodiment of the present disclosure is shown.

[0053] like Figure 2 As shown, the cache device 2000 includes a receiving module 210, a scheduling module 220, and an execution module 230.

[0054] The receiving module 210 is configured to receive multiple requests.

[0055] The scheduling module 220 includes multiple waiting queues.

[0056] The scheduling module 220 is configured to store the multiple requests into multiple waiting queues based on their request types. For example, these multiple waiting queues include a read / write request queue and a computation request queue.

[0057] The execution module 230 includes multiple execution pipelines. These multiple execution pipelines are coupled to corresponding wait queues in multiple wait queues. These multiple execution pipelines include read / write pipelines and computation pipelines.

[0058] The read / write pipeline is configured to execute read / write requests emitted from the read / write request queue.

[0059] The computation pipeline is configured to execute computation requests emitted from the computation request queue.

[0060] For example, the cache device 2000 can be used in general-purpose graphics processing units (GPGPUs) for last-level cache (LLC), global cache, etc., to provide high-bandwidth data access services for computing cores and to support computational operations at the cache level.

[0061] For example, the receiving module 210 can be coupled to an on-chip bus (e.g., a bus interface connecting the computing core and the cache).

[0062] For example, coupling refers to the direct connection between two modules or the indirect connection through an intermediate device to achieve the transmission of signals or data.

[0063] For example, the receiving module 210 can be configured to receive multiple requests from the computing core or other master device.

[0064] For example, the scheduling module 220 is coupled to the receiving module 210 and is used to obtain requests from the receiving module 210 and perform scheduling management.

[0065] For example, the scheduling module 220 is configured to store multiple requests into multiple waiting queues based on request type. For example, the request type can refer to the operation category of the memory access request, such as read request, write request, atomic operation request, reduction calculation request, etc., and the embodiments of this disclosure do not limit this.

[0066] For example, an atomic operation is an operation that cannot be interrupted and can be used for synchronization in a multi-threaded environment, such as atomic add and atomic compare and swap, which will not be interrupted by other operations during execution.

[0067] For example, a reduction operation is an operation that combines multiple data into one data by some operation (such as summation or finding the maximum value). For example, in GPGPU, it can be used for data aggregation in parallel computing.

[0068] For example, the scheduling module 220 parses the requests provided by the receiving module 210 and identifies the operation type of each request. For example, when the request to be processed is identified as a read request or a write request, the request is stored in the read / write request queue; when the request to be processed is identified as an atomic operation request or a reduction calculation request, the request is stored in the calculation request queue. The read / write request queue and the calculation request queue are independent of each other, so that different types of requests can be queued separately without affecting each other.

[0069] In one example, the request type can include read / write and computation types. Accordingly, multiple wait queues include read / write request queues and computation request queues. For example, the read / write request queue stores requests that require a read or write operation. Similarly, the computation request queue stores requests that require a computation operation, such as an atomic operation or a reduction operation.

[0070] For example, a read / write request queue can refer to a waiting queue used to store read requests and / or write requests, and can be implemented, for example, using registers, static random access memory or other storage circuits, which is not limited by the embodiments of this disclosure.

[0071] For example, a computation request queue can refer to a waiting queue used to store atomic operation requests and / or reduction computation requests, and is independent of the read / write request queue.

[0072] It should be noted that those skilled in the art can further subdivide the queue type according to actual design requirements, such as dividing the read / write request queue into a read request queue and a write request queue, or dividing the computation request queue into an atomic operation queue and a reduction operation queue. These variations all fall within the scope of this disclosure.

[0073] For example, scheduling module 220 can also be configured to schedule the issuance of requests in the queue. In each clock cycle, scheduling module 220 can select ready requests from the read / write request queue and / or compute request queue according to a preset scheduling strategy (e.g., priority scheduling, round-robin scheduling, or resource availability-based scheduling), and issue the selected requests to the corresponding execution pipeline in execution module 230.

[0074] In one implementation, the scheduling module 220 can employ a priority scheduling strategy, such as prioritizing read / write requests before processing computation requests, or dynamically scheduling requests based on factors such as request waiting time and resource availability. Within each clock cycle, the scheduling module 220 can select eligible requests from one or more waiting queues and dispatch them to the corresponding execution pipeline in the execution module 230.

[0075] For example, execution module 230 is coupled to scheduling module 220. For instance, multiple execution pipelines in execution module 230 are coupled to corresponding waiting queues in multiple waiting queues, meaning each execution pipeline is specifically designed to handle request types from its corresponding waiting queue. This one-to-one coupling effectively resolves resource contention issues between different types of requests, improving processing efficiency.

[0076] For example, a read / write pipeline can be coupled to a read / write request queue, and the read / write pipeline can be configured to execute read / write requests emitted from the read / write request queue.

[0077] For example, a read / write pipeline can be a multi-stage sequential logic circuit used to complete read or write operations over multiple clock cycles. For instance, a read / write pipeline can employ a multi-stage pipeline structure, including address decoding, hit detection, data read, and data write stages. In one implementation, the read / write pipeline can be further subdivided into read and write pipelines to handle read and write requests respectively.

[0078] For example, a compute pipeline can be coupled to a compute request queue, and the compute pipeline can be configured to execute compute requests emitted from the compute request queue.

[0079] For example, a computational pipeline can be another type of multi-stage sequential logic circuit used to complete computational operations over multiple clock cycles.

[0080] For example, a computation pipeline may include an integrated arithmetic logic unit (ALU) capable of performing specified computations (such as addition, subtraction, bitwise operations, comparisons, etc.) on data read from a cache line and operands carried in a request, and writing the computation result back to the cache line or returning it to the request source. For example, the depth of a computation pipeline may be greater than that of a read-write pipeline to accommodate the additional execution cycles required for computational operations.

[0081] For example, the request source can refer to the module that initiates the memory access request, such as the computing core in the GPGPU or other main device, and the embodiments of this disclosure do not limit this.

[0082] In one implementation, the receiving module 210 can receive memory access requests from the on-chip bus and pass them to the scheduling module 220. The scheduling module 220 can store the requests into a read / write request queue or a computation request queue based on the type of each request (e.g., by parsing the command field of the request). When a ready read or write request exists in the read / write request queue, the scheduling module 220 can issue the request to the read / write pipeline, which will then perform the read or write operation on the cache array. Similarly, for example, when a ready computation request exists in the computation request queue, the scheduling module 220 can issue the request to the computation pipeline, which will read the source data from the cache array, perform the computation operation, and write the result back. For example, "ready" means that the resources the corresponding request depends on (such as cache line data, write buffer space) are ready and meet the corresponding issuance conditions.

[0083] In one implementation, the output of the receiving module 210 can be coupled to the input of the scheduling module 220 to pass received requests to the scheduling module 220. The output of the read / write request queue of the scheduling module 220 can be coupled to the input of the read / write pipeline of the execution module 230 to send read / write requests to the read / write pipeline. The output of the computation request queue of the scheduling module 220 can be coupled to the input of the computation pipeline of the execution module 230 to send computation requests to the computation pipeline. The output of the execution module 230 can be coupled to the on-chip bus or other downstream modules to return read data or write responses.

[0084] The high-speed cache device 2000 provided in at least one embodiment of this disclosure can be applied to various scenarios requiring high-bandwidth, low-latency cache access, and is particularly suitable for the last-level cache in GPGPU chips. In an exemplary implementation, the number of waiting queues in the scheduling module 220 and the number of pipelines in the execution module 230 can be expanded according to actual needs. For example, the read / write request queue can be further divided into read request queues and write request queues, and read pipelines and write pipelines can be added accordingly; multiple independent computation request queues and computation pipelines can also be set up for different types of computational operations (such as atomic addition, atomic comparison swap, reduction summation, etc.) to achieve finer-grained parallel scheduling.

[0085] In at least one embodiment of the cache device provided in this disclosure, by dividing the waiting queue in the scheduling module into a read / write request queue and a computation request queue according to the request type, and setting up independent execution pipelines coupled to each waiting queue in the execution module, decoupling and parallel processing of read / write requests and computation requests are achieved. This architecture allows read / write operations and computation operations to be scheduled and executed independently, effectively solving the performance degradation problem caused by mixed requests blocking each other in the same pipeline, and effectively improving the throughput of the cache device. In at least one embodiment, by setting up a dedicated computation pipeline for computation requests, tasks such as atomic operations and reduction calculations can be devolved to the cache level for execution without affecting the original read / write bandwidth, thereby alleviating the resource pressure on general-purpose registers and shared memory inside the computing core and improving the overall computing efficiency of the GPGPU.

[0086] In some embodiments of this disclosure, to further improve the processing efficiency of address-dependent requests and reduce redundant accesses to cache storage units, the read / write request queue and read / write pipeline can be further subdivided. For example, the read / write request queue may include a main read / write request queue and a secondary read / write request queue. Correspondingly, the read / write pipeline may include a main read / write request pipeline and a secondary read / write request pipeline.

[0087] For example, the scheduling module 220 can be further configured to: in response to the main read / write request queue transmitting a first read / write request, allocate a second read / write request with the same access address as the first read / write request to the sub-read / write request queue; or, in response to the main read / write request queue transmitting a first read / write request, the sub-read / write request queue including a third read / write request, and the third read / write request not being transmitted in the current clock cycle, not allocate it to the sub-read / write request queue. For example, the access address corresponding to the third read / write request is different from that of the first read / write request.

[0088] For example, launching refers to the process by which the scheduling module 220 retrieves a request from the waiting queue and sends it to the corresponding execution pipeline.

[0089] For example, "not send" means that a request fails to be sent to the corresponding execution pipeline within a certain clock cycle due to reasons such as resource conflicts or unmet dependencies.

[0090] For example, a resource conflict refers to a situation where multiple requests compete for the same hardware resource (such as the same memory bank or the same data bus) within the same clock cycle, resulting in them being unable to execute simultaneously.

[0091] For example, a bank is a logical unit of a cache storage array. Multiple banks can be accessed in parallel to improve bandwidth. For example, a bank conflict occurs when multiple requests simultaneously access different addresses or the same address within the same bank.

[0092] For example, a main read / write request queue is coupled to a main read / write request pipeline to store and issue main requests. Similarly, a secondary read / write request queue is coupled to a secondary read / write request pipeline to store and issue secondary requests that are address-dependent on the main requests.

[0093] For example, the depth of the read / write request subqueue can be 1, meaning it can only store one subqueue request, to simplify hardware design and provide pairing timeliness.

[0094] For example, in embodiments of this disclosure, the scheduling module 220 may be further configured to execute pairing logic for primary and secondary requests.

[0095] For example, the scheduling module 220 can be further configured to, in response to the main read / write request queue issuing a first read / write request (i.e., a main request), search in the request buffer (e.g., a temporary storage unit located between the receiving module 210 and the scheduling module 220, or a request cache unit set up inside the scheduling module 220) for a second read / write request that has the same access address as the first read / write request. If such a second read / write request exists, the scheduling module 220 can allocate the second read / write request to the secondary read / write request queue.

[0096] For example, "same access address" means that the cache line address requested by the second read / write request is exactly the same as that of the first read / write request. In GPGPU applications, multiple warps need to perform operations such as incrementing and updating the same memory address. For example, by pairing requests with the same address, subsequent execution modules can use data forwarding mechanisms to allow secondary requests to directly obtain data from the primary request, thereby avoiding repeated reads and writes to cache storage units.

[0097] In one implementation, after the primary request is issued, the secondary request can be issued in the next clock cycle (e.g., after a resource check). For example, if the secondary queue for read / write requests has a depth of 1, the secondary request can be temporarily stored to ensure it can be processed immediately after the primary request.

[0098] For example, scheduling module 220 can be further configured to respond to the primary read / write request queue issuing a first read / write request, and the secondary read / write request queue already contains a third read / write request, which is not issued in the current clock cycle. In this case, scheduling module 220 will not make a new allocation to the secondary read / write request queue. For example, the access address corresponding to the third read / write request is different from that of the first read / write request. This operating mode can, for example, prevent requests in the secondary queue from being incorrectly overwritten. For example, if there is already a secondary request waiting to be issued in the secondary queue, it means that the pairing of the previous primary request has not yet been completed. At this time, if a new primary request attempts to write its paired secondary request into the secondary queue, the original secondary request will be lost. Therefore, scheduling module 220 can only make a new allocation when the secondary queue is empty or when the requests in the secondary queue can be issued in the current cycle. If the secondary queue is not empty and the requests in it cannot be issued due to resource conflicts (e.g., storage unit access conflicts with other requests issued in the current cycle), then the new primary request will be issued separately without generating a corresponding secondary request.

[0099] It should be noted that the depth of the read / write request sub-queue is not limited to 1. In other embodiments, it can be set to a greater depth (e.g., 2 or 4) to accommodate more sub-requests, depending on actual hardware overhead and performance requirements. Correspondingly, the pairing logic requires a more complex control mechanism, such as maintaining a sub-request list for each main request and handling the order relationship between multiple sub-requests and the main request. In at least one embodiment of this disclosure, the design of a read / write request sub-queue depth of 1 achieves a good balance between hardware cost and performance improvement.

[0100] In the caching device provided in at least one embodiment of this disclosure, by further dividing the read / write request queue into a main queue and a secondary queue, and correspondingly setting a main pipeline and a secondary pipeline in the read / write pipeline, and having the scheduling module execute request pairing logic based on the same access address, the address correlation characteristics in the GPGPU load can be effectively utilized. For example, when a main request is issued, if there is a secondary request accessing the same address, it is assigned to the secondary queue, allowing the secondary request to be processed immediately after the main request, and reducing redundant read / write operations on the cache storage unit by utilizing a data forwarding mechanism. This significantly reduces the power consumption and latency of cache access and improves cache bandwidth utilization.

[0101] In some embodiments of this disclosure, the scheduling module 220 may be further configured to, after the main queue of read / write requests sends out the first read / write request, send out the second read / write request in response to the second read / write request passing the resource check.

[0102] For example, resource checking refers to the scheduling module 220 verifying the availability of the hardware resources (including but not limited to storage units, data paths, buffer spaces, etc.) required for a secondary request before deciding whether to issue the secondary request. The secondary request is only allowed to be issued when all necessary resources are idle or available.

[0103] In one implementation, resource checking may include checking for access conflicts to memory units. For example, in a cache device employing a multi-bank architecture (e.g., divided into Bank0 and Bank1 based on the parity of cache line addresses), multiple requests accessing different addresses of the same memory bank, or the same address of the same memory bank, within the same clock cycle may cause conflicts. For instance, if a primary request is accessing the read port of Bank0, and a secondary request attempts to access the write port of Bank0 within the same cycle, a resource conflict may occur. For example, the scheduling module 220 can pre-determine whether a secondary request conflicts with other requests (including primary requests, computation requests, etc.) issued in the current cycle by maintaining the occupancy status of each memory bank and its ports.

[0104] In another implementation, resource checks may also include checking for free space in write buffers (e.g., write data buffers (wdatabuffer)) or read return buffers (e.g., credit buffers (credit buffer)). For example, if the secondary request is a write request, it needs to be ensured that the target write buffer has enough space to temporarily store the write data; if the secondary request is a read request, it needs to be ensured that the read return buffer has enough space to store the read data to be returned.

[0105] In some embodiments of this disclosure, the secondary request can be issued after the primary request. For example, the secondary request may be issued in the next clock cycle after the primary request, provided that the secondary request passes the resource check. This allows the secondary request to utilize forwarded data generated by the primary request in the previous cycle (e.g., data stored in the forward line register), thereby implementing a data forwarding mechanism and reducing repeated access to cache storage units.

[0106] For example, if a secondary request fails the resource check in the current cycle, the scheduling module 220 can retain it in the read / write request secondary queue and continue to perform resource checks in subsequent clock cycles until the resource becomes available before issuing it. Since the read / write request secondary queue has a depth of 1, the secondary request will not be overwritten by other requests, which helps maintain the integrity of the pairing relationship.

[0107] For example, when performing launch scheduling, the scheduling module 220 can allocate resources according to a preset priority strategy (e.g., primary requests have higher priority than computation requests, and computation requests have higher priority than secondary requests). When a secondary request cannot be launched due to resource check failure, the scheduling module 220 can allow other non-conflicting requests (e.g., computation requests) to be launched in the current cycle to make full use of pipeline resources.

[0108] It should be noted that the specific content of resource checks can be adjusted according to the microarchitecture design of the cache device. For example, in more complex multi-bank, multi-port storage systems, resource checks can be extended to detect access conflicts across multiple banks and multiple ports (read ports, write ports, atomic operation ports). Furthermore, resource checks can also include checks on data dependencies, such as whether a secondary request is waiting for the primary request's data to be ready.

[0109] In another implementation, resource checks can be performed by an independent resource arbitration unit in the scheduling module 220, which monitors the status of each execution pipeline and storage unit in real time and allocates resource authorization for each request to be launched.

[0110] In the caching device provided in at least one embodiment of this disclosure, by performing resource checks before the secondary request is issued and allowing issuance only when resources are available, pipeline stalls or execution errors caused by resource conflicts are effectively reduced, enabling the secondary request to safely and efficiently utilize the forwarded data generated by the primary request, thereby realizing the data forwarding function and reducing redundant access to the cache storage unit.

[0111] In some embodiments of this disclosure, the read / write request sub-queue can be configured to send a second read / write request (i.e., a sub-request) within at least one operating cycle (e.g., a clock cycle, i.e., within at least one clock cycle) after the read / write request main queue sends out a first read / write request (i.e., a main request).

[0112] For example, "within at least one clock cycle" can mean that there is at least one full clock cycle between the launch time of the secondary request and the launch time of the primary request, but not exceeding the preset maximum number of waiting cycles. In other words, the secondary request cannot be launched within the same clock cycle as the primary request, but can be launched in one or more subsequent clock cycles, depending on resource availability and scheduling policies.

[0113] For example, the execution of a secondary request might depend on intermediate data generated by the primary request (such as data read from a cache line or write data to be written). For instance, in a hardware implementation, the primary request's data may undergo several stages of processing in the main pipeline before being written to the forward line register for use by the secondary request. If the secondary request is emitted in the same cycle as the primary request, it will not receive ready-to-forward data, thus losing the benefits of pairing.

[0114] In one implementation, the read / write request sub-queue can be configured to attempt to issue a sub-request in the next clock cycle after the primary request is issued. If the sub-request passes resource checks in the next cycle (e.g., no memory access conflicts with other requests in the current cycle, forwarding register data is ready, etc.), it is issued in that cycle. This pattern of issuing the primary request and then issuing the sub-request in the next cycle minimizes the total processing latency of paired requests.

[0115] The inventors of this disclosure have also noted that, in some cases, a secondary request may fail to be launched in the next clock cycle due to resource conflicts. For example, suppose that in the current cycle, a primary request accesses the read port of Bank0, while a secondary request needs to access the write port of Bank0, but the write port is occupied by another computation request in the same cycle. In this case, the secondary request cannot be launched in the next cycle. To solve this problem, for example, a read / write request sub-queue can be configured to allow secondary requests to wait for launch for multiple clock cycles after the primary request is launched. For example, the secondary request can be retained in the read / write request sub-queue (assuming the read / write request sub-queue depth is 1) and continuously undergo resource checks in each subsequent clock cycle. Once the resource check passes, the secondary request can be launched. This reduces the possibility of pairing failure due to momentary resource conflicts, thereby improving the robustness of the system.

[0116] In one implementation, a maximum waiting period (e.g., 4 or 8 clock cycles) can be set for the secondary request. If the secondary request still cannot be issued after reaching the maximum waiting period, the scheduling module 220 can take one of the following measures: remove the secondary request from the secondary queue and reassign it to the main read / write request queue for normal processing as an independent request (i.e., abandon pairing and execute in a normal pipeline manner); or force the secondary request to be issued, even if resource conflicts exist, and handle the conflicts through pipeline pauses or bypass mechanisms; or trigger a pipeline refresh and reschedule the relevant requests. By setting a timeout mechanism, the long-term occupation of secondary queue resources by a single secondary request can be reduced, which helps to improve the overall throughput of the scheduling module.

[0117] For example, the timing configuration for issuing secondary requests within at least one clock cycle after the main read / write request queue issues a main request can also work in conjunction with the resource checking mechanism provided in the aforementioned embodiments. For instance, the issuance conditions for a secondary request may include two elements: first, a timing condition, i.e., at least one clock cycle has elapsed since the main request was issued; second, a resource condition, i.e., the secondary request passes the resource check. The secondary read / write request queue can be configured so that a secondary request can only be issued when both conditions are met simultaneously.

[0118] In the caching device provided in at least one embodiment of this disclosure, by configuring the secondary request to be issued within at least one clock cycle after the primary request is issued, the timing prerequisites required by the data forwarding mechanism (i.e., the data in the forwarding register is ready) are satisfied, and a flexible timing window is provided for the issuance of the secondary request. When the secondary request cannot be issued immediately in the next cycle due to resource conflicts, it is allowed to wait for the resources to be released before being issued in subsequent cycles, thereby reducing pairing failures caused by instantaneous resource contention. This design maximizes the performance benefits brought by the pairing mechanism (i.e., reducing redundant access to cache storage units) while ensuring data correctness.

[0119] Figure 3 A schematic block diagram of another cache device provided in at least one embodiment of the present disclosure is shown.

[0120] In some embodiments of this disclosure, such as Figure 3 As shown, the cache device 2000 may further include a storage module 240. The execution module 230 may include a write data buffer and a credit buffer.

[0121] The main flow of read and write requests can be configured as follows: in response to the first read / write request being a read request, access storage module 240 to read the target data and provide the target data to the credit buffer; or, in response to the first read / write request being a write request, retrieve the source data from the write data buffer and write it to storage module 240.

[0122] In one implementation, storage module 240 may employ a multi-bank architecture to improve parallel access capabilities. For example, storage module 240 may include a first storage unit (e.g., Bank0) and a second storage unit (e.g., Bank1).

[0123] For example, the first and second storage units can be partitioned based on the parity of cache line addresses. For instance, cache lines with even addresses can be stored in Bank0, and cache lines with odd addresses can be stored in Bank1, or vice versa. This parity partitioning allows requests with consecutive addresses to access different storage banks in parallel, thereby reducing the probability of storage bank conflicts. Those skilled in the art will understand that the number of storage banks in storage module 240 is not limited to two and can be expanded to more storage banks (e.g., 4, 8, or 16) according to design requirements, with each storage bank having an independent read / write port.

[0124] For example, a write data cache can be used to temporarily store write data to be written to storage module 240. In one implementation, the write data cache can be implemented using a register array or static random access memory, with multiple entries, each of which can store the source data of a write request and related control information (such as the destination address, byte enable, etc.).

[0125] For example, credit caching can refer to a return data buffer that uses a credit flow control mechanism. Before a request is sent, the available space (credit) in the buffer needs to be checked to prevent data overflow, thereby achieving lossless data transmission.

[0126] For example, a credit cache can be used to temporarily store data read from storage module 240 that is to be returned to the requesting source. The depth of the credit cache can be configured according to the system's requirements for return path latency and bandwidth, and the embodiments of this disclosure do not limit this.

[0127] For example, the main watershed for read and write requests can be configured to perform the appropriate operation based on the type of the first read or write request (i.e., the main request).

[0128] For example, the read / write request mainline can be configured to respond to the first read / write request as a read request, access storage module 240 to read the target data, and provide the target data to the credit cache.

[0129] For example, the read / write request mainline can be configured to respond to the first read / write request as a write request, whereby the read / write request mainline retrieves the source data from the write data cache and writes it to the storage module 240.

[0130] For example, storage module 240 can be coupled to the main read / write request pipeline, the sub-read / write request pipeline, and the compute pipeline, respectively, to provide data read / write services. For example, the input of the write data cache is coupled to the request receiving path or scheduling module to receive and temporarily store the source data of the write request; the output of the write data cache is coupled to the main read / write request pipeline to provide the source data during write operations. For example, the input of the credit cache is coupled to the main read / write request pipeline, the sub-read / write request pipeline, and the compute pipeline to receive read data returned by each pipeline; the output of the credit cache is coupled to the request return module or the on-chip bus to return data to the request source.

[0131] For example, the capacity and implementation of the write data cache and credit cache can be adjusted according to system performance requirements. For instance, in high-bandwidth application scenarios, the depth of the write data cache can be increased to accommodate more pending write requests, or the depth of the credit cache can be increased to support a larger pending read response queue. Furthermore, the number of storage banks in storage module 240 can be expanded according to cache capacity and parallelism requirements. Correspondingly, the address mapping strategy can be expanded from simple parity partitioning to more complex hash mapping to balance the access load across storage banks.

[0132] In the caching device provided in at least one embodiment of this disclosure, a clear data path is provided for the main pipeline of read and write requests by setting up an independent storage module (adopting a multi-storage structure), a write data cache, and a credit cache. During a read request, the main pipeline reads the target data from the storage module and sends it to the credit cache, achieving an efficient read return path; during a write request, the main pipeline retrieves the source data from the write data cache and writes it to the storage module, achieving decoupled processing of write data.

[0133] In some embodiments of this disclosure, the execution module 230 further includes a forward line register.

[0134] The main watershed for read / write requests can be further configured to write the target or source data into the forward line register in response to the first read / write request being either a read or write request.

[0135] The read / write request subpipeline can be further configured to: read first data from the forward line register in response to a second read / write request being a read request; or, obtain source data from the forward line register in response to both the second read / write request and the first read / write request being a write request, merge it with the write data of the second read / write request, and write the merged data to the storage module.

[0136] For example, a forwarding register can be a temporary storage unit used to hold cached line data involved in read / write operations during a main request. In one implementation, the width of the forwarding register can be configured to be equal to the size of a cached line (e.g., 32 bytes, 64 bytes, or 128 bytes) to allow it to fully store the data of a cached line. For example, the forwarding register can be implemented using a register array to achieve low-latency read / write access.

[0137] For example, the input of the forwarding register can be coupled to the main pipeline of read / write requests to receive target data (read requests) or source data (write requests) generated during the processing of the main request. For example, the output of the forwarding register can be coupled to the secondary pipeline of read / write requests to provide forwarded data to the secondary requests. Furthermore, in one implementation, the forwarding register can also be coupled to a write data buffer to retrieve the source data of the main request in a paired scenario where both the main and secondary requests are write requests.

[0138] For example, the main read / write request pipeline can be further configured to write the target data or source data into the forwarding register 5 in response to the first read / write request (i.e., the main request) being either a read request or a write request.

[0139] For example, when the primary request is a read request, the main stream reads the target data from storage module 240. In addition to providing this target data to the credit cache (as described above) for returning the request source, the main stream also writes the target data to forwarding register 5. This way, when a secondary request accesses the same address as the primary request, it can directly retrieve data from forwarding register 5, avoiding further access to storage module 240.

[0140] For example, when the primary request is a write request, the main stream of the read / write request retrieves the source data from the write data cache, preparing to write it to storage module 240. Simultaneously or before writing to storage module 240, the main stream writes this source data to forwarding register 5. This allows subsequent secondary requests (especially write or read requests accessing the same address as the primary request) to directly utilize this forwarded data.

[0141] For example, the read / write request sub-pipeline can be further configured to respond to a second read / write request (i.e., a sub-request) as a read request, whereby the read / write request sub-pipeline reads first data from forwarding register 5 and returns the first data as the read request's return data.

[0142] For example, the read / write request sub-pipeline can be further configured to, in response to a second read / write request being a write request and in response to a first read / write request being a write request, obtain source data (i.e., the write data of the main request) from forwarding register 5, merge it with the write data of the second read / write request, and write the merged data to storage module 240.

[0143] For example, at least one embodiment of this disclosure can employ a merged write mechanism. The secondary request obtains the source data of the primary request from the forwarding register 5, and then merges it with its own write data (e.g., byte-level merging via bitmasking) into a complete data set. The merged data is then written to the storage module 240 in one go. In this way, what originally required two write operations (one write operation for the primary request and one read, modify, and write operation for the secondary request) is simplified to a single write operation (the secondary request writes the merged data), while eliminating the need for either the primary request's write operation or the secondary request's read operation.

[0144] For example, reading, modifying, and writing refer to the process of first reading the original data in the storage unit, performing modification operations, and then writing the modified data back to the storage unit.

[0145] For example, merging can refer to the process of combining the write data from the main request with the write data from the secondary request, byte-by-byte enabled, to generate a complete cache line. For example, byte enable can refer to a control signal used to indicate which bytes are valid and which remain unchanged during a write operation.

[0146] In one implementation, the merging operation can be performed by merging logic circuitry in a secondary pipeline. This merging logic circuitry can receive two inputs, for example, one from the forwarding register (write data requested by the primary pipeline) and the other from the write data buffer (write data requested by the secondary pipeline). For example, based on the requested byte enable signal, the merging logic can selectively merge the two data streams byte by byte to generate the final data to be written. After merging, the secondary pipeline writes the merged data to the corresponding memory bank of the storage module 240.

[0147] It should be noted that the number of forwarding registers is not limited to one. In one implementation, multiple forwarding registers can be set up (e.g., forming a forwarding register queue) according to pipeline depth and concurrency requirements to support concurrent forwarding between multiple main requests and their corresponding sub-requests. Furthermore, the granularity of the merging logic can be adjusted according to the byte enable precision of the write request, supporting finer-grained data merging (such as byte-level, half-word-level, or word-level merging).

[0148] In the caching device provided in at least one embodiment of this disclosure, efficient data forwarding between primary and secondary requests is achieved by setting a forwarding register in the execution module and having the main pipeline write the target data of a read request or the source data of a write request into the register, while the secondary pipeline retrieves data from the forwarding register according to the type of the secondary request. For secondary read requests, data can be read directly from the forwarding register, eliminating one read operation from the storage module; for secondary write requests (where the primary request is also a write request), the source data of the primary request can be retrieved and merged with its own write data before being written to the storage module in one go, eliminating the write operation of the primary request and the read operation of the secondary request. This fully utilizes the address correlation between primary and secondary requests, effectively reducing redundant accesses to the storage module, thus reducing cache access power consumption and shortening request processing latency. In the ideal scenario of write + write pairing, what originally required two write operations (or one read, modify, and write operation) is simplified to one merged write operation, significantly improving processing efficiency and freeing up storage module bandwidth resources for other requests, significantly improving the overall performance of the caching device.

[0149] In some embodiments of this disclosure, the execution module 230 further includes multiple computing units and an arithmetic logic unit buffer (ALU buffer).

[0150] The computation pipeline can be further configured to: retrieve the first source data from the arithmetic logic unit buffer (ALU buffer), enter the corresponding computation unit among multiple computation units according to the operation type corresponding to the computation request, perform the corresponding computation operation, and write the corresponding computation result to the storage module 240.

[0151] For example, the arithmetic logic unit cache can be used to temporarily store operands (i.e., first source data) carried by a computation request. These operands can be immediate values ​​from the request source, or values ​​in registers or memory addresses specified by the request source.

[0152] In one implementation, the arithmetic logic unit cache can be implemented using a register array, which can have multiple entries, each corresponding to a computation request to be executed. Its depth can be configured according to the system's requirements for the concurrency of computation requests, and the embodiments of this disclosure do not limit this.

[0153] For example, multiple computational units may include different types of arithmetic logic units, such as atomic operation units and reduction operation units. For instance, atomic operation units can be used to perform operations such as atomic addition, atomic subtraction, and atomic comparison swaps. For instance, reduction operation units can be used to perform reduction operations such as summation, finding the maximum value, and finding the minimum value. Those skilled in the art will understand that the types of computational units disclosed herein are not limited to the two types mentioned above, and can be extended to more types according to design requirements, such as bitwise operation units and logical operation units.

[0154] For example, the input of the arithmetic logic unit cache 6 can be coupled to the scheduling module 220 to receive operands carried by the computation request. In one implementation, when a computation request enters the computation request queue, its operands can be stored along with the request, or read from the write data cache and sent to the arithmetic logic unit cache 6 at issue. The output of the arithmetic logic unit cache 6 is coupled to the computation pipeline to provide the first source data during the execution of the computation operation.

[0155] For example, multiple computing units can be coupled to the control logic of the computing pipeline, and the computing pipeline selects the corresponding computing unit to perform the computation based on the operation type of the computation request. For example, the input terminals of multiple computing units 7 can be coupled to the arithmetic logic unit cache 6 and the storage module 240 to receive first source data and second source data read from the storage module; for example, the output terminals of multiple computing units 7 can be coupled to the storage module 240 to write the computation results back to the storage module.

[0156] For example, the computation pipeline can be further configured to: retrieve the first source data from the arithmetic logic unit cache 6, enter the corresponding computation unit in multiple computation units 7 according to the operation type corresponding to the computation request, perform the corresponding computation operation, and write the corresponding computation result to the storage module 240.

[0157] In one implementation, for example, when a computation request is issued from the computation request queue to the computation pipeline, the computation pipeline obtains the operation type of the request (e.g., by parsing the command field of the request) and the target address. The computation pipeline determines the storage bank where the target cache line is located based on the target address (e.g., determining whether to access Bank0 or Bank1 based on address parity) and issues a read request to the storage module 240 to read the second source data (i.e., the original data in the cache line) stored in that cache line. Simultaneously, the computation pipeline retrieves the first source data (i.e., the operand) corresponding to the request from the arithmetic logic unit cache 6. For example, the computation pipeline can send the first and second source data to the corresponding computation unit 7 based on the operation type.

[0158] For example, if the operation type is atomic add, the data is sent to the atomic operation unit; if the operation type is reduce sum, the data is sent to the reduce operation unit. After the computation unit 7 performs the computation operation, it generates the computation result. For example, the computation pipeline can write the computation result to the corresponding cache line in the storage module 240. For example, a write operation can first read the original data, perform the computation, and then write back the result. For example, for a storage system that supports atomic operations, reading, computation, and writing can also be completed in one operation to reduce the interference of other requests in the intermediate state.

[0159] In one implementation, the computation pipeline can support multi-cycle computation operations. For example, for certain computational units (such as division, square root, etc.), the computation may require multiple clock cycles to complete. In this case, the computation pipeline can include a pipeline pause mechanism or employ independent multi-cycle computational units to prevent them from blocking the processing of other requests.

[0160] For example, the number and type of computing units can be expanded according to the actual computing needs of the GPGPU chip, and the embodiments disclosed herein are not limited in this regard. For example, computing units specifically for floating-point operations, computing units for vector operations, etc., can also be added. Accordingly, the width of the arithmetic logic unit cache 6 can also be adjusted according to the bit width of the operands (e.g., expanded from 32 bits to 64 bits or 128 bits). In addition, the computing pipeline can also support the pipelining of multi-cycle computing operations, that is, to achieve parallel processing among multiple computing units to improve computing throughput.

[0161] The caching device provided in at least one embodiment of this disclosure achieves the ability to directly perform atomic operations and reduction calculations at the cache level by setting an arithmetic logic unit cache and multiple computation units in the execution module, and by having the computation pipeline select the corresponding computation unit to perform the computation according to the requested operation type. The arithmetic logic unit cache provides dedicated operand storage space for computation operations, decoupling operands from request control information and simplifying pipeline design.

[0162] In some embodiments of this disclosure, in order to fully implement the processing flow of computation requests, the computation pipeline can be further configured to read raw data from the storage module 240 and perform operations with the operands carried in the request, thereby realizing computation tasks that require cached raw data, such as atomic operations or reduction computations.

[0163] For example, the computation pipeline can be further configured to read second source data from storage module 240 and perform computation operations on the second source data and the first source data.

[0164] For example, during the execution of a computation request in the computation pipeline, in addition to retrieving the first source data (i.e., the operands carried in the request) from the arithmetic logic unit cache 6, it is also necessary to read the data currently stored in the target cache line from the storage module 240, i.e., the second source data. The second source data is another input to the computation operation, and its value depends on the actual content of the cache line corresponding to the target address when the computation request arrives.

[0165] As previously described, in some embodiments of this disclosure, the storage module 240 employs a multi-bank structure to improve parallel access capabilities. For example, the first storage unit can be denoted as Bank0, and the second storage unit can be denoted as Bank1. The first and second storage units can be partitioned based on the parity of cache line addresses. For instance, cache lines with even addresses can be stored in Bank0 (i.e., the first storage unit), and cache lines with odd addresses can be stored in Bank1 (i.e., the second storage unit), or vice versa. This parity partitioning method allows requests with consecutive addresses to access different storage banks in parallel, thereby reducing the probability of storage bank conflicts.

[0166] It should be noted that the number of storage cells in storage module 240 is not limited to two, and can be expanded to more storage cells (e.g., 4, 8, or 16) according to design requirements. The embodiments disclosed herein do not impose any limitations on this. In addition, each storage cell can also have an independent read / write port.

[0167] To support parallel access across multiple pipelines, each memory unit can be further divided into independent read and write ports. For example:

[0168] The random access read port (ram read bank0) can be used to represent the read port of the first memory cell (Bank0), used to read data from Bank0;

[0169] The random access read port (ram read bank1) can be used to represent the read port of the second memory unit (Bank1), used to read data from Bank1;

[0170] The random access write port (ram write bank0) can be used to represent the write port of the first memory cell (Bank0), and is used to write data to Bank0;

[0171] The random access write port (ram write bank1) can be used to represent the write port of the second storage unit (Bank1), which is used to write data to Bank1.

[0172] By separating read and write ports, multiple pipelines can access different memory banks or different ports of the same memory bank within the same clock cycle, thereby reducing resource conflicts. For example, the main pipeline for read and write requests can simultaneously read data through RAM read bank0, while the compute pipeline writes data through RAM write bank1, without interference between the two.

[0173] For example, the main watershed for read and write requests can be configured to access storage units through the corresponding ramread bank (for read requests) or ram write bank (for write requests) based on the parity of the requested address. For instance, if the target cache line is stored in Bank0, it is accessed through ram read bank0 or ram write bank0; if it is stored in Bank1, it is accessed through ram read bank1 or ram write bank1.

[0174] For example, the read / write request subpipeline can be configured not to directly access the read port of the storage module (when using forwarded data), or to write merged data through the corresponding ram write bank when needed.

[0175] For example, the computation pipeline can be configured to retrieve the first source data from the arithmetic logic unit cache 6, read the second source data through the corresponding ram read bank according to the parity of the cache line number, enter the corresponding computation unit to perform the computation according to the operation type, and write the computation result to the storage module through the corresponding ram write bank according to the parity of the cache line number.

[0176] In one implementation, after issuing a computation request, the computation pipeline can decode the target address of the request to determine the memory bank where the target cache line is located. For example, based on the parity of the cache line number, it can determine whether to access ramread bank1 or ramread bank0. Subsequently, the computation pipeline sends a read request to the storage module 240 to read the data stored in that cache line. This read operation and the operation of retrieving the first source data from the arithmetic logic unit cache 6 can be performed in parallel, or they can be executed step by step according to the pipeline design.

[0177] For example, in executing a computational operation, after acquiring the first source data and the second source data, the computation pipeline can send these two data streams to the corresponding computation unit 7 to perform the computational operation, based on the operation type of the computation request. The type of computation unit 7 depends on the operation type; for example, an atomic operation unit is used to perform atomic operations, and a reduction operation unit is used to perform reduction computations.

[0178] For example, when writing computation results, after the computation operation is completed, the computation pipeline can write the computation result to the corresponding cache line in storage module 240. In one implementation, the write operation writes the computation result to ram write bank1 or ram write bank0 based on the parity of the cache line number, overwriting the original second source data. After the write operation is completed, the computation pipeline can optionally send the computation result to the credit cache simultaneously for return to the request source. For example, for atomic operations, it may be necessary to return the original value before the operation (i.e., the second source data) or the result value after the operation, which may depend on the semantics of the request.

[0179] For example, the computation pipeline can access the storage module 240 using a read, modify, and write operation flow. Exemplarily, the computation pipeline can read second source data from the RAM read bank of the storage module 240; the computation unit performs the computation operation; and the computation result is written to the RAM write bank of the storage module 240. Read and write operations can be performed separately through different ports of the storage module 240 (RAM read bank and RAM write bank).

[0180] For example, the first source data (e.g., from the arithmetic logic unit cache 6) and the second source data (e.g., from the storage module 240) play different roles in the computation operation. For example, the first source data may come from the request source and is the external input to the computation operation; the second source data may be the value currently stored in the cache line and is the basic data for the computation operation. The order of computation and the way they are combined are determined by the operation type. For example, in an addition operation, the two are symmetrical, while in a comparison-swap operation, the first source data is used as the comparison value and the second source data is used as the value to be compared.

[0181] It should be noted that the granularity of reading the second source data can be adjusted according to the needs of the computation operation. For byte-level or half-word-level atomic operations, the computation pipeline can read only the byte or half-word corresponding to the target address, instead of the entire cache line, to reduce the access bandwidth consumption of the storage module. Correspondingly, when writing the computation result, byte-enabled control should also be used to update only the target byte or half-word.

[0182] The cache device provided in at least one embodiment of this disclosure allows the computation pipeline to read second source data from the RAM readbank of the storage module, perform computation operations with first source data cached from the arithmetic logic unit, and then write the computation result to the storage module through the RAM writebank, thus fully realizing the execution of atomic operations and reduction computation at the cache level. By obtaining the two source data required for the computation operation from the operand carried in the request (first source data) and the original data of the cache line (second source data) respectively, the computation pipeline can complete the computation task independently without relying on the computation core. The separate design of the RAM readbank and RAM writebank allows read and write operations to be performed through different ports, simplifying the control logic of the storage module.

[0183] In some embodiments of this disclosure, in order to reasonably allocate pipeline resources among multiple waiting queues, the scheduling module can be configured to request launch according to a preset priority strategy.

[0184] For example, the scheduling module 220 can be further configured to prioritize the main queue of read / write requests over the computation request queue, and prioritize the computation request queue over the sub-queue of read / write requests.

[0185] For example, launch priority refers to the order in which the scheduling module 220 selects requests for launch from multiple waiting queues within the same clock cycle. For example, when multiple queues have ready requests simultaneously, the scheduling module 220 can select requests in descending order of priority, prioritizing the launch needs of high-priority queues. For example, in some embodiments of this disclosure, the priorities from high to low are: main queue for read / write requests, queue for computation requests, and sub-queue for read / write requests.

[0186] In some embodiments of this disclosure, the priority strategy can work in conjunction with the aforementioned resource checking mechanism. For example, when selecting a request according to priority, the scheduling module 220 also needs to verify whether the selected request passes the resource check (e.g., whether there is a memory conflict with a request already issued in the current cycle). If a request in a high-priority queue fails the resource check, the scheduling module 220 can skip the request and continue checking the next priority queue.

[0187] In one implementation, scheduling module 220 can simultaneously issue multiple requests within each clock cycle (e.g., primary and computation requests are issued simultaneously, or primary and secondary requests are issued in different cycles), provided that all requests pass resource checks and do not exceed the parallel processing capacity of execution module 230. A priority strategy determines the selection order when multiple requests are ready simultaneously, rather than restricting the issuance of only one request.

[0188] In at least one embodiment of this disclosure, the primary queue for read / write requests has the highest priority because the primary request initiates the pairing process, and its issuance can trigger the pairing and allocation of secondary requests. If the issuance of the primary request is delayed, secondary requests may not be able to pair in time, reducing the efficiency of the data forwarding mechanism. The secondary queue for computation requests has the next highest priority. Computation requests, for example, involve atomic operations or reduction computations, requiring a longer processing time and potentially occupying the read and write ports of the storage module. Prioritizing computation requests, while ensuring that the primary request can be issued in time, can fully utilize computation pipeline resources and prevent computational tasks from piling up. The secondary queue for read / write requests has the lowest priority because secondary requests depend on the forwarded data generated by the primary request, and their issuance sequence is naturally after the primary request. Furthermore, assuming the depth of the secondary queue for read / write requests is 1, secondary requests can be issued in the next clock cycle after the primary request is issued, so a high priority is not required. Setting the secondary queue for read / write requests to the lowest priority can reduce the amount of scheduling resources occupied by secondary requests, thus reducing their impact on the processing of primary and computational requests.

[0189] It should be noted that the priority strategy can be adjusted according to the application scenario and performance requirements. For example, in compute-intensive applications, the priority of the compute request queue can be appropriately increased; in memory-intensive applications, the high priority of the main queue can be maintained. The embodiments of this disclosure do not impose any restrictions on this. In addition, the priority strategy can adopt a dynamic adjustment mechanism to adaptively schedule according to the occupancy depth or request waiting time of each queue. The embodiments of this disclosure do not impose any restrictions on this.

[0190] In some embodiments of this disclosure, in order to perform cache request hit detection and cache line filling when a cache miss occurs, such as Figure 3 As shown, the cache device 2000 may further include a hit detection module 260.

[0191] For example, the hit detection module 260 can also be configured to, in response to a target read / write request in multiple requests hitting a target cache line in the storage module, adjust the status information of the target cache line according to the request type of the multiple requests, and provide the target read / write request to the scheduling module; or, in response to a target read / write request in multiple requests not hitting a target cache line in the storage module, provide the target read / write request to the scheduling module.

[0192] For example, the scheduling module 220 can be further configured to send a read request to memory to obtain the target data corresponding to the target read / write request and write it to the target cache line in response to the target read / write request not hitting the target cache line in the storage module.

[0193] For example, a memory hit means that the data requested by the memory access request exists in the cache storage module. A memory miss means that the data requested by the memory access request does not exist in the cache storage module and needs to be read from memory.

[0194] For example, the target cache line refers to the cache line accessed by the request. In the case of a hit, it is an existing cache line, and in the case of a miss, it is a cache line to be allocated.

[0195] For example, the hit detection module 260 can be configured to perform hit detection on target read / write requests among multiple requests and perform corresponding operations based on the detection results.

[0196] For example, in one implementation, when a target read / write request hits a target cache line, the hit detection module 260 needs to update the cache line's status information based on the request type. For instance, if the request is a read request, the hit detection module 260 can mark the cache line's status as "recently accessed" for subsequent cache replacement algorithm decisions; if the request is a write request, the hit detection module 260 can mark the cache line's status as "dirty," indicating that the data in the cache line has been modified but not yet written to memory. This status information can be stored in a tag storage unit associated with the storage module 240. After the status information is adjusted, the hit detection module 260 passes the target read / write request to the scheduling module 220, awaiting subsequent issuance and execution.

[0197] For example, status information refers to management information related to cache lines, such as valid bit, dirty bit, and recently accessed information, which are used to support cache consistency management and replacement strategies.

[0198] For example, in one implementation, when a target read / write request misses, the hit detection module 260 does not need to perform state adjustment (because the target cache line does not exist or has not been allocated), and directly passes the request to the scheduling module 220, which handles the allocation and filling of cache lines in the case of a miss.

[0199] For example, the scheduling module 220 can be further configured to send a read request to the memory to obtain the target data corresponding to the target read / write request and write it to the target cache line in response to the target read / write request not hitting the target cache line in the storage module 240.

[0200] For example, in one implementation, when the scheduling module 220 receives a missed read / write request, it first needs to allocate an available cache line (i.e., the target cache line) for the target request. If all current cache lines are occupied, the scheduling module 220 can select a line to replace according to a preset replacement strategy. If the cache line to be replaced is dirty (i.e., the data has been modified but not yet written to memory), the scheduling module 220 also needs to write the dirty data back to memory before filling the new cache line. For example, after the cache line allocation is completed, the scheduling module 220 sends a read request to memory (e.g., off-chip dynamic random access memory) to request the reading of the complete data block corresponding to the target cache line. After memory returns the data, the scheduling module 220 writes the data to the allocated cache line and marks the cache line as valid. At this time, the cache line data on which the original request depends is ready, and the scheduling module 220 can send the request to the corresponding waiting queue (e.g., a read / write request queue or a computation request queue) to wait for execution.

[0201] For example, a cache replacement strategy can refer to an algorithm for selecting a cache line to be replaced when the cache is full and a cache line needs to be allocated for a new request, such as Least Recently Used (LRU) or Pseudo-Least Recently Used (PLRU). The embodiments disclosed herein do not limit this.

[0202] For example, in one implementation, the hit detection module 260 can update the status of cache lines while performing hit detection, so that the request has the correct status information when it enters the scheduling module 220, which simplifies the processing logic of the scheduling module.

[0203] For example, the hit detection module 260 can further integrate the control logic of the cache replacement algorithm. For example, when a cache miss occurs and replacement is needed, the hit detection module 260 can be responsible for selecting the cache line to be replaced and triggering a write-back operation for the dirty data. For example, the hit detection module 260 can also support parallel access from multiple ports, handling hit detection for multiple requests simultaneously to meet the needs of high-bandwidth scenarios.

[0204] The caching device provided in at least one embodiment of this disclosure achieves standardized management of the cache access process by setting a hit detection module to perform hit detection and status updates before the request enters the scheduling module. When a hit occurs, the hit detection module adjusts the cache line status information according to the request type, providing an accurate basis for cache consistency management and replacement strategies; when a miss occurs, the scheduling module is responsible for sending a read request to memory and filling the cache line, enabling the request to obtain the required data and continue execution.

[0205] In some embodiments of this disclosure, to support out-of-order processing and flexible scheduling of requests, the scheduling module 220 may further include a request cache. The request cache may be configured to temporarily store at least one request that has been hit.

[0206] In one implementation, the request cache can take the form of an out-of-order request buffer.

[0207] For example, after the hit detection module 260 completes the hit detection of the request, it can pass the request to the scheduling module 220 and temporarily store it in the request cache. The request waits in the request cache until the resources it depends on (such as cache line data, write buffer space, etc.) are ready, and then it is dispatched to the corresponding waiting queue (such as the main queue of read and write requests, the secondary queue of read and write requests, or the computation request queue) to wait for launch.

[0208] In one implementation, the request cache can take the form of an out-of-order request buffer.

[0209] For example, "out-of-order" means that the order in which requests are stored in the buffer is independent of the order in which they enter the buffer, and the order in which requests are selected from the buffer is also independent of their storage order.

[0210] For example, the input of the request cache can be coupled to the output of the hit detection module 260 to receive requests that have passed the hit detection. The output of the request cache can be coupled to the inputs of the main read / write request queue, the secondary read / write request queue, and the computation request queue to distribute ready requests to the corresponding waiting queues. The scheduling control logic inside the scheduling module 220 is coupled to the request cache to manage the write, wake-up, and read operations of requests. For example, wake-up refers to the process of updating the status of a request from not ready to ready, which can be triggered, for example, by a resource ready signal.

[0211] For example, in one implementation, after the hit detection module 260 completes processing the request, it sends the request and its related information (e.g., request type, target address, operands, cache line status, etc.) to the scheduling module 220. The scheduling module 220 allocates a free entry in the request cache for storage and records the resource information that the request depends on (e.g., if the request misses, it depends on cache line data returned from memory; if the request hits but cache line data is not yet ready, it depends on the ready status of the cache line data). For example, when the resource that the request depends on becomes ready (e.g., memory returns cache line data, or cache line data is read from the storage module), the corresponding entry in the request cache is marked as ready. For example, the scheduling control logic of the scheduling module 220 scans the ready entries in the request cache every clock cycle and distributes them to the corresponding waiting queue (read / write request main queue, read / write request sub-queue, or computation request queue) according to the request type. After distribution, the request cache entry is released to store new requests.

[0212] It should be noted that the number of entries in the request cache can be configured according to the system's requirements for the number of concurrent pending requests. A larger number of entries allows for the processing of more pending requests simultaneously, but also increases hardware overhead. In one implementation, the request cache can support dependency detection of requests, such as detecting address dependencies between requests, to support the aforementioned primary / secondary request pairing mechanism.

[0213] In at least one embodiment of the caching apparatus provided in this disclosure, the scheduling module, by setting a request cache (implemented using an out-of-order request buffer), can temporarily store multiple requests that have undergone a hit detection and support out-of-order request processing. When some requests need to wait for memory data to return due to cache misses, other ready requests can skip these waiting requests and be preferentially dispatched to the waiting queue and executed, thereby reducing pipeline stalls caused by partial request blocking. This out-of-order processing capability effectively hides memory access latency and improves the flexibility of the scheduling module and pipeline utilization.

[0214] In some embodiments of this disclosure, the scheduling module 220 may be further configured to write the data in the target cache line into memory before operating on the target cache line according to the target read / write request, in response to the data in the target cache line being modified and not yet written to memory.

[0215] For example, in a cache device, each cache line can be associated with a status bit to indicate whether the data in that cache line is consistent with the data in memory. For instance, when new data is written to a cache line but has not yet been written back to memory, the cache line is called a dirty cache line, and its corresponding status bit is set to "dirty".

[0216] For example, a dirty data write-back operation is triggered when the scheduling module 220 processes a missed read / write request and needs to allocate an available cache line (i.e., the target cache line) for the target request. During the allocation process, if the selected cache line is currently in a dirty state, the scheduling module 220 writes the existing dirty data in the cache line back to memory before writing new data to the cache line.

[0217] For example, in one implementation, the scheduling module 220 can be configured to select a cache line to be replaced according to a preset cache replacement strategy. This cache line can be an idle cache line or a cache line currently occupied by other data. For example, the scheduling module 220 checks the status bit of the selected cache line. If the dirty status bit of the cache line is not set (i.e., the cache line is clean), subsequent cache line filling operations can be performed directly. For example, if the dirty status bit of the cache line is set (i.e., the cache line is dirty), the scheduling module 220 initiates a dirty data write-back process. For example, the scheduling module 220 can send a write request to memory to write the complete data of the cache line to the corresponding address in memory. This write request can include the target memory address (obtained from the cache line's label) and the data in the cache line. For example, before the write-back operation is completed, the cache line is considered occupied and cannot be used for other requests. After the write-back operation is completed, the scheduling module 220 can clear the dirty status bit of the cache line and then continue with the cache line filling operation.

[0218] For example, in some embodiments of this disclosure, the dirty data write-back operation can occur before the cache line filling operation. For instance, for a missed read / write request, the scheduling module 220 first completes the allocation of the target cache line and the dirty data write-back (if necessary), then sends a read request to memory to obtain the new data required by the target request, and writes the new data to the allocated target cache line. This sequence can maintain the correctness of the data in memory and reduce the risk of data loss due to the failure to write back dirty data in a timely manner.

[0219] For example, dirty data write-back operations and sending new data read requests to memory can be performed in parallel. For example, scheduling module 220 can start preparing for new data read operations after sending a write-back request without waiting for the write-back to complete, but it must ensure that the write-back operation has been completed and the cache line has been cleared when the new data is written to the cache line.

[0220] For example, the hit detection module 260 is responsible for updating the status information of the cache line when a hit is requested, including marking the cache line as dirty when a write hit occurs. When the cache line is subsequently replaced, the scheduling module 220 can determine whether a write-back operation needs to be performed based on the status bits set by the hit detection module 260.

[0221] For example, dirty data write-back operations can be optimized using a write-back buffer. For instance, dirty data to be written back can be temporarily stored in a write-back buffer, with independent write-back logic responsible for sending write requests to memory, thereby reducing the main processing flow of the blocking scheduling module 220. Furthermore, for storage modules supporting multiple storage banks, dirty cache line write-backs from different storage banks can be performed in parallel, and the embodiments of this disclosure do not impose any limitations on this.

[0222] In at least one embodiment of the caching apparatus provided in this disclosure, when the scheduling module allocates a cache line for a target read / write request, if the selected cache line is in a dirty state, it writes its data back to memory before operating on the cache line. This dirty data write-back mechanism maintains data consistency between the cache and memory, reducing the risk of data loss errors caused by dirty data being overwritten. In at least one embodiment, by placing the dirty data write-back operation before the cache line filling operation, the latest data copy is always stored in memory, providing a correct data foundation for subsequent cache line filling and request execution.

[0223] In some embodiments of this disclosure, such as Figure 3 As shown, the cache device 2000 also includes a request return module 270.

[0224] The request response module 270 can be configured to provide a corresponding request response to the target request source via the on-chip bus in response to the completion of the execution of the target request among multiple requests.

[0225] For example, in one implementation, after a pipeline in the execution module 230 (e.g., the main pipeline for read / write requests, the sub-pipeline for read / write requests, or the computation pipeline) completes the processing of the request, it sends the execution result to the request return module 270.

[0226] For example, the content of the execution result depends on the type of request:

[0227] For example, for a read request, the execution result includes the target data read from storage module 240 (or forwarded data obtained through forwarding register 5), and request return module 270 returns the data to the request source via on-chip bus.

[0228] For example, for a write request, the result can be a write response (e.g., a write completion confirmation signal). The request return module 270 returns this response to the request source via the on-chip bus to notify the request source that the write operation has been completed.

[0229] For example, for a computation request (such as an atomic operation or a reduction computation), the execution result may include the original value before computation or the result value after computation. For example, depending on the semantics of the request, the request return module 270 returns the result to the request source via the on-chip bus.

[0230] For example, the input of the request return module 270 can be coupled to each pipeline of the execution module 230 (main read / write request pipeline, sub-read / write request pipeline, and computation pipeline) to receive the execution results output by each pipeline. In one implementation, the credit cache in the execution module 230 can serve as a relay station for execution results, and the request return module 270 reads the data to be returned from the credit cache. The output of the request return module 270 is coupled to the on-chip bus to send the request response to the request source.

[0231] For example, the request return module 270 can support multiple responses being returned simultaneously. Since multiple pipelines of the execution module 230 can process requests in parallel, multiple requests may complete execution within the same clock cycle or consecutive clock cycles. The request return module 270 can aggregate these completion events and return the responses in an appropriate order via the on-chip bus.

[0232] In one implementation, the request-return module 270 may internally include a return buffer for temporarily storing completed but not yet returned responses. For example, when the on-chip bus is available, the request-return module 270 retrieves the response from the return buffer and sends it to the bus. The depth of the return buffer can be configured according to the system's requirements for return latency and throughput, and the embodiments of this disclosure do not limit this.

[0233] For example, the request-return module 270 can support multiple on-chip bus protocols, such as Advanced Extensible Interface (AXI), Advanced High Performance Bus (AHB), or other custom bus protocols. The request-return module 270 can package and format the returned data according to the requirements of the bus protocol to ensure compatibility with different bus standards.

[0234] Figure 4 A schematic diagram illustrating the application of a cache device provided in at least one embodiment of the present disclosure is shown.

[0235] like Figure 4 As shown, the high-speed cache device includes a first receiving module 310, a first hit detection module 360, a first scheduling module 320, a first execution module 330, a cache storage unit 380, and a memory 390.

[0236] For example, the first receiving module 310 can be configured to receive various requests from the GPGPU computing core or other on-chip modules, including read / write requests, atomic operation requests, and reduction calculation requests, and synchronously transmit the requests to the first hit detection module 360.

[0237] For example, the first hit detection module 360 ​​can be configured to query the tag array of the cache storage unit 380 to determine whether the requested address hits the cache line: if it hits, the request and hit status information are forwarded to the first scheduling module 320; if it misses, the first scheduling module 320 is notified to prepare to send a missing request to the memory 390, wait for the memory 390 to return data and fill it into the cache storage unit 380, and then continue to process the original request.

[0238] For example, the first scheduling module 320 is a unit that implements multiple launches. Internally, it maintains three types of request queues: a main read / write queue, a computation queue, and a secondary read / write queue, which correspond to the three parallel pipelines of the first execution module 330, respectively. The first scheduling module 320 can be configured to classify requests after hit detection: read / write requests enter the main read / write queue, and atomic / reduction computation requests enter the computation queue; at the same time, it matches the addresses of ready requests in the main queue. If there are requests with the same address waiting to be processed, it is assigned to the secondary read / write queue, forming a main-secondary request pairing relationship, preparing for subsequent data forwarding.

[0239] For example, the first scheduling module 320 can be configured to send requests according to the priority of the main read / write queue over the compute queue and the compute queue over the secondary read / write queue, and perform resource conflict detection before sending to ensure that the maximum of 3 requests sent in a single cycle (from the three queues respectively) will not cause access conflicts to the multiple storage banks (e.g., Bank0 / Bank1) or port resources of the cache storage unit 380, thus ensuring the feasibility of pipeline parallel execution.

[0240] For example, the three queues of the first execution module 330 and the first scheduling module 320 are coupled one-to-one to form three independent execution pipelines: a main read / write pipeline, a computation pipeline, and a secondary read / write pipeline. For requests from the main read / write pipeline, the main read / write pipeline can be configured to access the corresponding memory bank of the cache storage unit 380 based on the parity of the cache line number: read requests read data from the target memory bank and temporarily store it in the credit buffer; write requests obtain data from the write data buffer and write it to the target memory bank, while simultaneously writing the read and write data to the forwarding register to provide data forwarding support for the secondary pipeline. For requests from the computation pipeline, the computation pipeline can be configured to obtain operands from the computation operand buffer (ALU buffer), combine them with the original data read from the corresponding memory bank of the cache storage unit 380, complete the computation in the atomic / reduction ALU, and write the result back to the corresponding memory bank of the cache storage unit 380.

[0241] For example, for pairing requests in the read / write sub-queue, the read / write sub-pipeline can be configured to directly obtain the cached line data of the main request from the forwarding register: read requests do not need to access cache storage unit 380 again and directly read the forwarded data; write requests merge their own write data with the write data of the main request and write it to cache storage unit 380, thereby eliminating redundant cache read / write operations and improving bandwidth utilization. If the first scheduling module 320 detects a cache miss, the first execution module 330 can cooperate to initiate a miss request to memory 390. The data returned by memory 390 is received by the first execution module 330 and filled into cache storage unit 380 before continuing to complete the execution of the original request.

[0242] For example, the first execution module 330 can be configured to extract the results of completed requests from each buffer of the first execution module 330, including read request return data, write request completion response and calculation request result, and return them to the request source module through the on-chip bus. At the same time, the return rate is coordinated through a credit flow control mechanism to avoid data overflow at the receiving end.

[0243] For example, cache storage unit 380, as the storage entity of LLC, can adopt a multi-storage structure to cooperate with the first execution module 330 to achieve parallel access, maintain the status of cache lines such as valid bits and dirty bits, and when cache lines are replaced or written back, the first execution module 330 will write dirty data back to memory 390 to ensure data consistency. Memory 390, as system-level storage, provides missing data filling and dirty data write-back services for cache storage unit 380, and works with the first scheduling module 320 and the first execution module 330 to complete the entire process of LLC memory access and computation processing.

[0244] Figure 5 A schematic diagram illustrating the application of another cache device provided in at least one embodiment of the present disclosure is shown.

[0245] Figure 5 The interaction structure between the scheduling module and the execution module in the LLC pipelined multi-issue device in some embodiments of this disclosure is shown, which realizes the operational logic of request splitting, multi-queue scheduling, multi-pipeline parallel execution and data forwarding.

[0246] For example, requests from the hit detection module can first be stored in a request cache (e.g., an out-of-order request buffer) to temporarily store various requests that have completed address hit detection and cache line allocation. The request cache can use multiplexing logic to divert requests to three independent waiting queues according to the request type and address dependency: read / write requests, including read and write operations, are allocated to the main read / write request queue; requests with computational functions, such as atomic operations and reduction calculations, are allocated to the computation request queue; the secondary read / write request queue does not actively accept new requests, but only when there are ready requests to be issued in the main read / write request queue, the scheduling logic matches a pending read / write request with the same access address as the main request, and then allocates the request with the same address to the secondary read / write request queue, forming a main-secondary request pairing relationship.

[0247] For example, the scheduling module can be configured to perform corresponding resource checks on ready requests in the three queues when executing a request launch operation in each clock cycle, and only launch the request to the corresponding execution pipeline after all corresponding resource checks have passed.

[0248] For example, the scheduling module can use a preset priority scheduling rule to set the launch priority of the main queue of read / write requests to be higher than that of the computation request queue, and set the launch priority of the computation request queue to be higher than that of the sub-queue of read / write requests. For example, within the same clock cycle, hardware resources are allocated to requests that pass the resource check in the high priority queue first. If a ready request in the high priority queue fails the resource check, then the resource check and launch operation are performed on the ready requests in the next priority queue in turn.

[0249] For example, the scheduling module can reduce resource contention by enabling multiple requests (up to three requests from three queues) issued within a single clock cycle to avoid concurrent access to the same memory bank or the same hardware port of the storage module through resource checks.

[0250] For example, the resource checks that pending requests in the main queue of read and write requests need to pass include: write data cache resource check, credit cache resource check, storage module read port resource check, and storage module write port resource check.

[0251] For example, the resource checks that a request to be issued in the computation request queue must pass include: arithmetic logic unit cache resource checks, credit cache resource checks, storage module read port resource checks, storage module write port resource checks, atomic operation unit resource checks, and reduction operation unit resource checks.

[0252] For example, the resource checks that pending requests in the read / write request subqueue must pass include: write data cache resource checks, credit cache resource checks, and storage module write port resource checks.

[0253] For example, requests that pass resource checks can be executed in parallel through three independent execution pipelines.

[0254] For example, requests from the main read / write request queue enter the main read / write pipeline, which is configured with four types of data paths. For write requests, the source data is retrieved from the write data cache and written to the corresponding storage. For read requests, the read port of the corresponding storage is accessed according to the parity of the cache line number, the target data is read and stored in the credit cache. At the same time, during the read / write operation, the main read / write pipeline will synchronously write the target data to be read and the source data to be written to the forwarding register, providing data multiplexing support for the secondary read / write pipeline.

[0255] For example, when a request from the computation request queue enters the computation pipeline, it first retrieves the operands carried by the request from the arithmetic logic unit cache, and at the same time reads the original data of the cache line from the read port of the corresponding memory. According to the operation type corresponding to the request, it is sent to the atomic operation unit or reduction operation unit to perform the corresponding computation operation. After the computation is completed, the computation result is written back to the cache through the write port of the corresponding memory, and the result is stored in the credit cache for flow control management according to the request type (such as if data needs to be returned).

[0256] For example, requests from the secondary read / write request queue enter the secondary read / write pipeline as paired requests with the primary request at the same address. This pipeline optimizes execution through a data forwarding mechanism. For secondary read requests, cache line data is read directly from the forwarding register of the primary read / write pipeline without needing to access the read port of the storage module again. For secondary write requests, the write data is merged with the write data of the primary request and written to the cache in one go through the write port of the storage module, avoiding duplicate read / write operations for requests at the same address.

[0257] For example, the processing results of the three pipelines are output through their respective interfaces. The processing results of the main read / write pipeline are transmitted through the first interface, the processing results of the computation pipeline are transmitted through the second interface, and the processing results of the secondary read / write pipeline are transmitted through the third interface. All processing results are finally aggregated to the request return module, which provides the corresponding request response to the request source through the on-chip bus, thus completing the entire process of LLC request processing.

[0258] Figure 6 A schematic diagram illustrating the application of another cache device provided in at least one embodiment of the present disclosure is shown.

[0259] Figure 6 The data paths, timing control, and module interaction logic of the three execution pipelines in the LLC pipelined multi-issue device in some embodiments of this disclosure are shown, presenting the collaborative workflow of the main read / write pipeline, the computing pipeline, the secondary read / write pipeline, the cache storage unit, the forwarding register, and various execution units.

[0260] For example, requests from the three queues of the scheduling module are input to the corresponding pipelines through the first interface, the second interface, and the third interface, respectively: the first interface corresponds to the main read / write pipeline of the main read / write request queue, the second interface corresponds to the computation pipeline of the computation request queue, and the third interface corresponds to the secondary read / write pipeline of the secondary read / write request queue.

[0261] For example, in the main read / write pipeline (first interface), requests input to the first interface are first routed to four core data paths. Write data is temporarily stored in a write data buffer, and then processed through multiple register delay stages (…). Figure 6 After the pipeline registers (with the middle triangular symbol) are synchronized, they are connected to the write ports of the first and second memory units to complete the write operation to the corresponding cache memory. The read ports of the first and second memory units select Bank0 (first memory unit) or Bank1 (second memory unit) to access based on the parity of the cache line of the requested address. The cache line data read is sent to the credit cache for temporary storage after passing through the selector, and is simultaneously written to the forwarding register to provide data forwarding support for the secondary pipeline. The arithmetic logic unit cache temporarily stores the auxiliary data associated with the main request and is synchronously connected to the forwarding register to ensure the data consistency between the main and secondary requests. All data paths are pipelined through multi-level register delay stages, splitting long-path operations into multi-cycle execution, improving the system clock frequency and parallelism.

[0262] For example, in a computation pipeline (second interface), computation requests input to the second interface are first stored in the arithmetic logic unit cache, temporarily storing the operands and address information. After a delay stage, they are merged with the original data from the cache line of the storage unit read port, and then allocated to the atomic operation unit or the reduction operation unit through a selector. The atomic operation unit handles atomic requests (such as comparison and swap, addition atomic operations), and the execution result is written back to the corresponding storage bank through the storage unit write port after passing through multiple delay stages. The reduction operation unit handles reduction requests (such as summation, finding extreme values), and adopts a multi-stage pipelined reduction operation unit structure to break long-latency reduction calculations into multi-cycle pipeline execution, improving throughput. After the calculation is completed, the result is also written back to the storage unit through a delay stage. The calculation result is also synchronously sent to the credit cache, which works with the credit cache of the main read / write pipeline to achieve flow control and prevent data overflow at the receiving end when the request returns.

[0263] For example, in a secondary read / write pipeline (third interface), the secondary request input to the third interface is a paired request with the same address as the main pipeline. Its execution logic relies on the forwarding register of the main pipeline to achieve data multiplexing optimization. After a delay stage, the secondary read request directly reads the cached line data already acquired by the main pipeline from the forwarding register, sends it to the credit cache, and completes the read request processing without accessing the storage unit read port again. The secondary write request merges its own write data with the data in the write data cache of the main pipeline, and after a delay stage, writes it to the corresponding storage bank in one go through the storage unit write port, avoiding repeated read / write operations for requests with the same address. The write data path of the secondary pipeline shares the storage unit write port with the main pipeline. The scheduling module performs resource conflict detection in advance to ensure that the primary and secondary requests do not access the same write port or storage bank simultaneously within a single cycle, ensuring the feasibility of parallel execution.

[0264] For example, the first and second storage units constitute the LLC's dual-bank structure (Bank0 / Bank1), with separate read / write ports, supporting parallel access. All three pipelines can select the target storage bank based on the parity of the cache line address. The forwarding register is the core hub for data forwarding between the main and secondary pipelines. Cache line data read and write data from the main pipeline are written to this register in real time, while the secondary pipeline directly reads and reuses the data. This reduces redundant storage access for requests at the same address at the hardware level, significantly improving LLC bandwidth utilization. The latency-level design of all data paths enables the three pipelines to issue in parallel within a single cycle, with each stage operation synchronized in an orderly manner, achieving efficient multi-issue processing of LLC memory access and computation requests.

[0265] Figure 7 A schematic flowchart of a cache operation method provided in at least one embodiment of the present disclosure is shown.

[0266] like Figure 7 As shown, the cache operation method includes steps S400-S420. This cache operation method can be applied, for example, to the cache apparatus provided in any embodiment of this disclosure.

[0267] Step S400: Receive multiple requests.

[0268] Step S410: Based on the request types of the multiple requests, store the multiple requests into multiple waiting queues respectively.

[0269] Step S420: Execute read and write requests emitted from the read and write request queue via the read and write pipeline, and execute computation requests emitted from the computation request queue via the computation pipeline.

[0270] In some embodiments of this disclosure, step S410 may further include step S411 or step S412.

[0271] Step S411: In response to the main queue of read / write requests sending out the first read / write request, the second read / write request with the same access address as the first read / write request is assigned to the sub-queue of read / write requests.

[0272] Step S412: In response to the main read / write request queue transmitting the first read / write request, the secondary read / write request queue including the third read / write request, and the third read / write request not being transmitted in the current clock cycle, no allocation is made to the secondary read / write request queue. For example, the access address corresponding to the third read / write request is different from that of the first read / write request.

[0273] In some embodiments of this disclosure, step S411 may further include step S4111.

[0274] Step S4111: After the first read / write request is issued in the main read / write request queue, a second read / write request is issued in response to the resource check of the second read / write request.

[0275] In some embodiments of this disclosure, for example, the resource check in step S4111 may include step S41111.

[0276] Step S41111: Check whether there is an access conflict between the second read / write request and other requests issued in the current clock cycle for accessing the memory cell.

[0277] In some embodiments of this disclosure, step S411 may further include step S4112.

[0278] Step S4112: Within at least one clock cycle after the main read / write request queue sends out the first read / write request, send out the second read / write request from the secondary read / write request queue.

[0279] In some embodiments of this disclosure, step S420 may further include step S421 or step S422.

[0280] Step S421: In response to the first read / write request being a read request, access the storage module through the main waterline of the read / write request to read the target data and provide the target data to the credit cache.

[0281] Step S422: In response to the first read / write request being a write request, retrieve the source data from the write data cache through the main read / write request pipeline and write it to the storage module.

[0282] In some embodiments of this disclosure, for example, step S421 or step S422 may further include step S4211 or step S4212.

[0283] Step S4211: Through the main watershed of read / write requests, access the first or second storage unit according to the parity of the cache line number to execute the read request.

[0284] Step S4212: Access the first or second storage unit based on the parity of the cache line number through the main read / write request pipeline to execute the write request.

[0285] In some embodiments of this disclosure, steps S413-S415 may be included after step S411, for example.

[0286] Step S413: In response to the first read / write request being either a read request or a write request, write the target data or source data into the forwarding register via the main waterline of the read / write request.

[0287] Step S414: In response to the second read / write request being a read request, read the first data from the forwarding register through the read / write request sub-pipeline.

[0288] Step S415: In response to the second read / write request being a write request and in response to the first read / write request being a write request, obtain the source data from the forwarding register through the read / write request sub-pipeline, merge it with the write data of the second read / write request, and write the merged data into the storage module.

[0289] In some embodiments of this disclosure, step S420 may further include step S423.

[0290] Step S423: Retrieve the first source data from the arithmetic logic unit cache through the computation pipeline, and enter the corresponding computation unit among multiple computation units according to the operation type corresponding to the computation request to perform the corresponding computation operation, and write the corresponding computation result into the storage module.

[0291] In some embodiments of this disclosure, step S423 may further include step S4231, for example.

[0292] Step S4231: Read the second source data from the storage module through the computation pipeline, and perform computation operations on the second source data and the first source data.

[0293] In some embodiments of this disclosure, step S410 may further include step S413.

[0294] Step S413: Set the emission priority of the main read / write request queue to be higher than that of the compute request queue, and set the emission priority of the compute request queue to be higher than that of the secondary read / write request queue.

[0295] In some embodiments of this disclosure, steps S401-S403 may be included, for example, after step S400 and before step S410.

[0296] Step S401: In response to the target read / write request in the storage module hitting the target cache line among multiple requests, adjust the status information of the target cache line according to the request type of the multiple requests, and provide the target read / write request to the scheduling module.

[0297] Step S402: In response to the target read / write request not hitting the target cache line in the storage module among multiple requests, the target read / write request is provided to the scheduling module.

[0298] Step S403: In response to the target read / write request not hitting the target cache line in the storage module, a read request is sent to the memory through the scheduling module to obtain the target data corresponding to the target read / write request and write it to the target cache line.

[0299] In some embodiments of this disclosure, step S404 may be included, for example, after step S401 or step S402 and before step S410.

[0300] Step S404: Temporarily store at least one request that has passed the hit check in the request cache.

[0301] In some embodiments of this disclosure, step S403 may further include step S4031, for example.

[0302] Step S4031: In response to the fact that the data in the target cache line has been modified but has not yet been written to memory, the data in the target cache line is written to memory by the scheduling module before operating the target cache line according to the target read / write request.

[0303] In some embodiments of this disclosure, step S430 may be included after step S420, for example.

[0304] Step S430: In response to the completion of the target request among multiple requests, the corresponding request response is provided to the target request source via the on-chip bus through the request return module.

[0305] It should be noted that the specific functions and beneficial effects of each step in the cache operation method provided in any embodiment of this disclosure can be referred to the relevant descriptions of the cache device embodiments above, and will not be repeated here.

[0306] Figure 8 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0307] For example, such as Figure 8As shown, the first electronic device 700 includes at least one processor 701 and at least one memory 702. The at least one memory 702 includes one or more computer program modules. These computer program modules are stored in the memory 702 and configured to be executed by the at least one processor 701. The one or more computer program modules include instructions for performing data transfers from the aforementioned cache device, and when executed by the at least one processor 701, they can perform corresponding computational tasks. The memory 702 and the processor 701 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0308] For example, processor 701 may be a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), artificial intelligence (AI) accelerator, or other processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA), etc. It may include a cache device, as described in at least one embodiment of this disclosure, to enable data transmission between different clock domains. For example, the central processing unit (CPU) may be an x86, ARM, or RISC-V architecture. Processor 701 may be a general-purpose processor or a special-purpose processor, capable of controlling other components in the first electronic device 700 to perform desired functions.

[0309] For example, memory 702 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0310] Figure 9 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0311] The electronic devices in at least one embodiment of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0312] The electronic device includes at least one processor and a memory. The processor may be referred to as processing device 801 as described below, and the memory may include at least one of ROM 802, RAM 803, and storage device 808 as described below. The memory is used to store programs for performing the methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0313] like Figure 9 As shown, the second electronic device 800 may include a processing device 801 (e.g., a central processing unit (CPU), digital signal processor (DSP), image processor (GPU), general-purpose graphics processor (GPGPU), or other forms of processing unit with data processing capabilities and / or program execution capabilities), and may include a cache device according to any embodiment of this disclosure, for example, used as an LLC. Various programs and data required for the operation of the second electronic device 800 are also stored in RAM 803. The processing device 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interfaces are also connected to bus 804.

[0314] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, displays, speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows the second electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data.

[0315] Although Figure 9 A second electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0316] In particular, according to at least one embodiment of this disclosure, the process described above with reference to the flowchart can be implemented as a computer software program.

[0317] For example, at least one embodiment of this disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of at least one embodiment of this disclosure.

[0318] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0319] In at least one embodiment of this disclosure, a computer-readable storage medium can be any tangible medium that includes or stores a program that can be used or combined with an instruction execution system, apparatus, or device. In at least one embodiment of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or combined with an instruction execution system, apparatus, or device. The program code included on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0320] The aforementioned computer-readable medium may be included in the aforementioned second electronic device 800; or it may exist independently and not assembled into the second electronic device 800.

[0321] Figure 10 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0322] For example, such as Figure 10 As shown, a non-transitory computer-readable storage medium 900 stores computer-readable instructions 901, which, when executed by at least one processor, perform one or more steps of the cache operation method described above.

[0323] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard drive for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be... Figure 8 The memory 702 in the memory is described in the foregoing content and will not be repeated here.

[0324] Figure 11 A schematic block diagram of an artificial intelligence processor provided in at least one embodiment of the present disclosure is shown.

[0325] At least one embodiment of this disclosure also provides an artificial intelligence processor. For example... Figure 11 As shown, the artificial intelligence processor 1000 may include a first cache device 1100.

[0326] For example, the first cache device 1100 may include the cache device provided in any embodiment of this disclosure.

[0327] For example, Figure 1 The schematic diagram of the general-purpose graphics processing unit (GPGPU) shown can serve as an exemplary implementation of the artificial intelligence processor provided in the embodiments of this disclosure.

[0328] For example, the AI ​​processor 1000 may include multiple computing units (CUs), a thread block scheduling module, a thread block distribution module, and an on-chip interconnect bus. A first cache device 1100 is coupled to the multiple computing units and can be configured as the last-level cache (LLC) of the AI ​​processor 1000.

[0329] In one implementation, the AI ​​processor 1000 can be used to perform various AI computing tasks, such as training or inference of deep neural networks. During computation, warps within multiple computing units (e.g., streaming multiprocessors SM) generate numerous memory access requests and computation requests (e.g., reduction operations on neural network weights, atomic operations on gradient updates). For example, these requests are sent to a first cache device 1100 via an on-chip bus.

[0330] For example, the artificial intelligence processor 1000 provided in this disclosure embodiment may include various types of processor chips.

[0331] For example, the artificial intelligence processor 1000 may include any one of the following: a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), a deep learning processing unit (DPU), an accelerated processing unit (APU), and a general-purpose graphics processing unit (GPGPU).

[0332] It should be noted that, regardless of the specific architecture adopted, as long as it integrates the high-speed caching device provided in any embodiment of this disclosure as a component of its caching level (e.g., the last-level cache), it falls within the scope of this disclosure.

[0333] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the inventive concept of the present disclosure are within the scope of protection claimed by the present disclosure.

[0334] The following points should be noted regarding this disclosure:

[0335] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0336] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0337] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0338] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A cache apparatus, characterized by The cache device is configured to perform computational tasks at the cache level, and the cache device includes: The receiving module is configured to receive multiple requests; A scheduling module includes multiple waiting queues and is configured to store the multiple requests into the respective waiting queues based on their request types. The multiple waiting queues include a read / write request queue and a computation request queue. The execution module includes multiple execution pipelines, each coupled to a corresponding waiting queue in one of the multiple waiting queues, and includes read / write pipelines and computation pipelines. The read / write pipeline is configured to execute read / write requests emitted from the read / write request queue, and the compute pipeline is configured to execute compute requests emitted from the compute request queue.

2. The cache apparatus of claim 1, wherein, The read / write request queue includes a main read / write request queue and a secondary read / write request queue; the read / write pipeline includes a main read / write request pipeline and a secondary read / write request pipeline. The scheduling module is further configured as follows: In response to the main read / write request queue issuing a first read / write request, a second read / write request with the same access address as the first read / write request is allocated to the secondary read / write request queue; or, In response to the main queue of read / write requests transmitting a first read / write request, the sub-queue of read / write requests including a third read / write request, and the third read / write request not being transmitted in the current clock cycle, and not being allocated to the sub-queue of read / write requests, wherein the access address corresponding to the third read / write request is different from that of the first read / write request.

3. The cache device according to claim 2, characterized in that, The scheduling module is further configured to, after the main queue of read / write requests sends out the first read / write request, send out the second read / write request in response to the second read / write request passing the resource check.

4. The cache device according to claim 3, characterized in that, The resource check includes checking whether the second read / write request conflicts with other requests issued during the current clock cycle for access to memory units.

5. The cache device according to claim 2, characterized in that, The read / write request sub-queue is configured to send the second read / write request within at least one clock cycle after the read / write request main queue sends the first read / write request.

6. The cache device according to claim 2, characterized in that, The cache device also includes a storage module. The execution module includes a write data cache and a credit cache. The main watershed for read / write requests is configured as follows: In response to the first read / write request being a read request, the storage module is accessed to read the target data, and the target data is provided to the credit cache, or In response to the first read / write request being a write request, the source data is retrieved from the write data cache and written to the storage module.

7. The cache device according to claim 6, characterized in that, The storage module includes a first storage unit or a second storage unit. The read / write request pipeline is further configured to access the first storage unit or the second storage unit based on the parity of the cache line number to execute the read request or the write request.

8. The cache device according to claim 6, characterized in that, The execution module also includes a forwarding register. The main watershed for read / write requests is further configured to write the target data or the source data into the forwarding register in response to the first read / write request being a read request or a write request. The read / write request sub-pipeline is further configured as follows: In response to the second read / write request being a read request, first data is read from the forwarding register, or... In response to the second read / write request being a write request and in response to the first read / write request being a write request, the source data is obtained from the forwarding register, merged with the write data of the second read / write request, and the merged data is written to the storage module.

9. The cache device according to claim 2, characterized in that, The scheduling module is further configured such that the launch priority of the main read / write request queue is higher than that of the computation request queue, and the launch priority of the computation request queue is higher than that of the secondary read / write request queue.

10. The cache device according to claim 1, characterized in that, The execution module also includes multiple computation units and arithmetic logic unit caches. The computation pipeline is further configured to retrieve the first source data from the arithmetic logic unit cache, enter the corresponding computation unit among the plurality of computation units according to the operation type corresponding to the computation request, perform the corresponding computation operation, and write the corresponding computation result into the storage module.

11. The cache device according to claim 10, characterized in that, The computation pipeline is further configured to read second source data from the storage module and perform computation operations on the second source data and the first source data.

12. The cache device according to any one of claims 1-11, characterized in that, The cache device further includes: The storage module includes multiple cache lines; The hit detection module is configured to, in response to a target read / write request in the plurality of requests hitting a target cache line in the storage module, adjust the status information of the target cache line according to the request type of the plurality of requests, and provide the target read / write request to the scheduling module; or, in response to a target read / write request in the plurality of requests not hitting a target cache line in the storage module, provide the target read / write request to the scheduling module. The scheduling module is further configured to, in response to the target read / write request not hitting the target cache line in the storage module, send a read request to the memory to obtain the target data corresponding to the target read / write request, and write it into the target cache line.

13. The cache device according to claim 12, characterized in that, The scheduling module also includes: Request caching is configured to temporarily store at least one request that has been hit.

14. The cache device according to claim 12, characterized in that, The scheduling module is further configured to write the data in the target cache line into the memory before operating the target cache line according to the target read / write request, in response to the data in the target cache line being modified but not yet written to memory.

15. The cache device according to any one of claims 1-11, characterized in that, The cache device further includes: The request response module is configured to provide a corresponding request response to the target request source via the on-chip bus in response to the completion of the execution of the target request among the plurality of requests.

16. A cache operation method, applied to a cache device, characterized in that, The cache device is configured to perform computation tasks at the cache level. The cache device includes multiple execution pipelines and multiple wait queues. The multiple execution pipelines are respectively coupled to the corresponding wait queues in the multiple wait queues. The multiple execution pipelines include read-write pipelines and computation pipelines. The cache operation method includes: Receive multiple requests; Based on the request types of the multiple requests, the multiple requests are respectively stored in the multiple waiting queues, wherein the multiple waiting queues include a read / write request queue and a computation request queue; and The read / write pipeline executes read / write requests emitted from the read / write request queue, and the compute pipeline executes compute requests emitted from the compute request queue.

17. An electronic device, characterized in that, The electronic device includes the cache device according to any one of claims 1-15.

18. An artificial intelligence processor, characterized in that, The artificial intelligence processor includes the cache device as described in any one of claims 1-15.

Images