Protocol request execution method, cache, computing device and computing system

Abstract

The invention relates to a protocol request execution method, a cache, computing equipment and a computing system.The method is applied to the cache and comprises the steps that a target protocol request sent by a target computing unit is received; under the condition that the first source operand of the target protocol request does not hit the cache, distributing a corresponding target cache line for the target protocol request; the first step of reading the first source operand from the memory and the second step of reading a second source operand of the target protocol request from a target cache region corresponding to the target protocol request in the storage buffer region are executed in parallel, and the second source operand is written into a target cache line; under the condition that the first source operand is received, protocol operation matched with the target protocol request is executed on the first source operand and the second source operand, and a protocol operation result of the target protocol request is obtained; and returning a reduction operation result to the target calculation unit. Therefore, the preparation time of the two source operands of the protocol request can be realized, and the processing efficiency of the protocol request is improved.

Description

Technical Field

[0001] This disclosure relates to the technical field of caching, and more specifically, to a specification request execution method, a cache, a computing device, and a computing system. Background Technology

[0002] Computing devices, represented by Graphics Processing Units (GPUs), are widely used in general computing and have become core hardware in applications such as Artificial Intelligence (AI) and scientific computing. In these applications, the instructions and data executed by the program are typically stored in memory and read and manipulated by the computing units (or computing cores) within the computing device. As the processing speed of computing units far exceeds the access speed of memory, memory access has severely constrained computing performance. To alleviate this speed difference, computing devices typically integrate at least two levels of cache to temporarily store recently accessed instructions and / or data. Because the access latency of the cache is much lower than that of memory, the computing unit prioritizes retrieving instructions and / or data from the cache, thereby improving the processing efficiency of the computing device.

[0003] In multi-level cache architecture design, the Last Level Cache (LLC), as the last level of the on-chip cache system, is the key hub connecting the caches of each private computing core with the high-bandwidth memory outside the chip, and undertakes the core functions of multi-core data sharing, temporary caching, and data interaction scheduling.

[0004] In the actual computational process of artificial intelligence chips, the reduction operation is a fundamental core operation that runs through the entire model training and inference process. It is commonly found in various AI operators such as feature aggregation, gradient accumulation, softmax normalization, and batch normalization. The core requirement of this type of reduction operation is to aggregate the source operands obtained by multiple parallel computing cores into the last-level cache to complete the aggregation operation. This includes conventional reduction logic such as summation, averaging, finding the maximum value, and finding the minimum value, and finally outputting a single aggregation result for subsequent computing units to use.

[0005] The current implementation of last-level cache reduction operations suffers from insufficient dedicated temporary storage resources and limited parallel processing capabilities due to physical area constraints. This makes it unsuitable for the multi-core parallel, high-throughput, and low-latency computing requirements of AI chips. It not only reduces the execution efficiency of the reduction operator itself but also triggers a chain reaction, leading to a decrease in overall chip computing power utilization, increased power consumption, and increased latency. This makes it difficult to meet the stringent performance requirements of current large-scale, high-density AI computing scenarios. Summary of the Invention

[0006] One objective of this disclosure is to provide a new technical solution for performing reduction operations on cached data, thereby improving the execution efficiency of reduction operations.

[0007] According to a first aspect of this disclosure, a specification request execution method is provided for use with a cache, the method comprising: Receive a target specification request sent by the target computing unit; wherein the target specification request includes the address information of the first source operand and the second source operand; If the first source operand does not hit the cache, a corresponding target cache line is allocated for the target reduction request; The first step of reading the first source operand from memory based on the address information and the second step of reading the second source operand from the target cache area corresponding to the target reduction request in the storage buffer and writing the second source operand into the target cache line are executed in parallel. Upon receiving the first source operand, perform a reduction operation on the first source operand and the second source operand that matches the target reduction request to obtain the reduction operation result of the target reduction request; The reduction operation result is returned to the target computing unit.

[0008] Optionally, the method further includes: Identify whether the target cache line is in an idle state; If the target cache line is not in an idle state, it is determined whether the data in the target cache line is consistent with the corresponding data in the memory. If the data in the target cache line is consistent with the corresponding data in the memory, the target cache line is cleared.

[0009] Optionally, the method further includes: If the data in the target cache line is inconsistent with the corresponding data in memory, the data stored in the target cache line is written to the memory and the space of the target cache line is released.

[0010] Optionally, the method further includes: If the first source operand hits the cache, the second source operand is read from the target cache region, and the first source operand is read from the cache; a reduction operation matching the target reduction request is performed on the first source operand and the second source operand to obtain the reduction operation result.

[0011] Optionally, the method further includes: Write the result of the reduction operation into the target cache line.

[0012] Optionally, the method further includes: When the second source operand is written to the target cache line, the storage buffer is notified to release the target cache region.

[0013] Optionally, the method further includes: During the execution of the second step, other specification requests are executed in parallel.

[0014] According to a second aspect of this disclosure, a cache is also provided, the cache comprising: The request receiving circuit is configured to receive a target specification request sent by the computing unit; wherein the target specification request includes address information of a first source operand and a second source operand; The cache control circuit is configured to allocate a corresponding target cache line for the target specification request if the target specification request does not hit the cache. The memory access circuit is configured to perform a first step of reading a first source operand from memory that matches the target specification request based on the address information requested by the target specification request; The request execution circuit is configured to, when the memory access circuit executes the first step, execute the second step of reading the second source operand of the target reduction request from the storage buffer corresponding to the target reduction request and writing the second source operand into the target cache line; and, upon receiving the first source operand, perform a reduction operation on the first source operand and the second source operand to obtain the reduction operation result of the target reduction request. The result return circuit returns the reduction operation result to the computing unit.

[0015] Optionally, the cache is a last-level cache.

[0016] According to a third aspect of this disclosure, a computing device is also provided, the computing device comprising: Multiple computing units; A cache, wherein the cache is the cache according to the second aspect of this disclosure; or, the cache is configured to perform the method according to the first aspect of this disclosure; and, An on-chip bus is provided, wherein the computing unit is coupled to the on-chip bus, and the on-chip bus is coupled to the cache.

[0017] According to a fourth aspect of this disclosure, a computing system is also provided, the computing system including a control device and a computing device according to a third aspect of this disclosure, the control device being used to control the computing device to perform computing tasks.

[0018] Through the embodiments of this disclosure, during the execution of a reduction request, the first step of reading the first source operand of the target reduction request from memory and the second step of reading the second source operand of the target reduction request from the target cache area of ​​the storage buffer and writing the second source operand into the target cache line are executed in parallel. This breaks the limitation of the prior art where the reading and writing of the two source operands of a reduction request are executed serially, realizing the synchronous advancement of the two core data preparation steps and shortening the overall data preparation time of the reduction request. Compared with the serial execution mode that waits for one step to complete before starting another step, this embodiment does not require additional waiting time. While the memory access circuit reads the first source operand, the request execution circuit simultaneously completes the reading and writing of the second source operand, providing a foundation for the rapid initiation of subsequent reduction operation results and improving the overall processing efficiency of the reduction request.

[0019] The features and advantages of the embodiments of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of these embodiments.

[0021] Figure 1 This is a schematic diagram of the structure of a computing system to which the methods of the embodiments of this disclosure can be applied; Figure 2 This is a flowchart illustrating a specification request execution method according to some embodiments; Figure 3 This is a schematic diagram of the specification request execution flow according to some embodiments; Figure 4 This is a schematic diagram of the specification request execution flow according to some other embodiments; Figure 5 This is a schematic diagram of the architecture of a request execution circuit according to some embodiments; Figure 6 This is a schematic diagram of the cache structure according to some embodiments; Figure 7 These are schematic diagrams of the structure of a computing device according to some embodiments; Figure 8 This is a schematic diagram of the structure of a computing system according to some embodiments. Detailed Implementation

[0022] Various exemplary embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0023] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the embodiments of this specification or their application or use.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0025] This disclosure relates to a scheme for performing reduction operations on a cache, which dynamically adjusts the receiving speed of memory access requests to balance the consumption of cache processing resources by read and write requests. Figure 1 This is a schematic diagram of the structure of a computing system that can apply the specification operation execution scheme provided in the embodiments of this disclosure. For example... Figure 1 As shown, the computing system 100 may include a computing device 110, a control device 120, and a storage device 130, and the computing device, control device, and storage device may be connected to each other via a bus network or other means.

[0026] The storage device can be used to store instructions and / or data, which can be retrieved and used by a control device or a computing device. For example, the storage device can store program instructions executed by the control device or computing device, as well as data such as text, images, audio, and configuration parameters. The control device can control the computing device to execute related processes or tasks to achieve related system functions, such as artificial intelligence, scientific computing, or image processing functions based on user needs. Exemplarily, the control device can be a central processing unit (CPU); the computing device can be a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a neural network processing unit (NPU), or a tensor processing unit (TPU); and the storage device can be static random access memory (SRAM), read-only memory (ROM), or erasable programmable read-only memory (EPROM). This storage device can also be called memory, and this computing device can also be called a chip, processor, or artificial intelligence chip.

[0027] Continue as Figure 1As shown, the computing device 110 may include a computing unit 111 and a cache 112. The computing unit may be the core component in the computing device that performs computing tasks, and may also be referred to as a computing core, arithmetic core, or processing core. The cache may be used to cache instructions and / or data, and may also be referred to as a cache circuit, on-chip memory circuit, or on-chip memory.

[0028] This computing device contains multiple computing units, which can execute a process or task together or individually. Taking a GPU as an example, the computing unit can be a GPU Core, and a GPU consists of multiple GPU Cores.

[0029] The processing efficiency of the cache in this computing device is greater than that of external storage devices, but the storage capacity is smaller. This cache can be used to store recently used instructions and / or data to improve the processing efficiency of the computing device. For example, the computing device may include multi-level caches (at least two levels), such as a Level 1 cache (L1 Cache) and a Level 2 cache (L2 Cache), or a Level 1 cache (L1 Cache), a Level 2 cache (L2 Cache), and a Level 3 cache (L3 Cache). The Level 2 or Level 3 cache, acting as the Last Level Cache (LLC), is the Level 1 cache with the largest capacity and closest to the external storage device 130. It can directly access the storage device 130, for example, by reading instructions from or writing data to the storage device 130 via a bus. For example, each computing unit can have its own independent level-one cache, allowing each level-one cache to be used independently by a single computing unit. Alternatively, multiple computing units within the computing device can be grouped into multiple computing unit groups (e.g., SPCs), with each computing unit group having its own independent level-one cache, allowing each level-one cache to be shared by the computing units within that group. The last-level cache can be configured to be shared by all or some of the computing units in the computing device; this last-level cache can also be called the final level cache, shared cache, etc.

[0030] For example, a cache can employ a set-associative organization structure. For instance, sibling caches can be divided into multiple sets, each containing multiple ways, and each way containing one cache line (also called a cache block). In this way, a cache line can be uniquely identified by an index within the (set, way) group. A specific memory address in memory can be mapped to any cache line, thus improving processing efficiency when a computing unit needs to access a specific memory address by accessing the mapped cache line.

[0031] The current implementation of last-level cache reduction operations suffers from insufficient dedicated temporary storage resources and limited parallel processing capabilities due to physical area constraints. This makes it unsuitable for the multi-core parallel, high-throughput, and low-latency computing requirements of AI chips. It not only reduces the execution efficiency of the reduction operator itself but also triggers a chain reaction, leading to a decrease in overall chip computing power utilization, increased power consumption, and increased latency. This makes it difficult to meet the stringent performance requirements of current large-scale, high-density AI computing scenarios.

[0032] To address the above issues, this embodiment of the present disclosure performs the first step of reading the first source operand of the target specification request from memory and the second step of reading the second source operand of the target specification request from the target cache area of ​​the storage buffer and writing the second source operand into the target cache line in parallel during the execution of the specification request. This achieves the synchronous advancement of the two core data preparation steps and shortens the overall data preparation time of the specification request.

[0033] The following combination Figure 1 The computing system described herein illustrates various embodiments.

[0034] <First Embodiment> Figure 2 A flowchart illustrating a specification request execution method according to some embodiments is shown. This method is applied to... Figure 1 Cache 112. This cache 112 can be a last-level cache, or it can be a cache from another level that is shared by a large number of computing units. For example... Figure 2 As shown, the method of this embodiment may include the following steps S210 to S250.

[0035] Step S210: Receive a target specification request sent by the target computing unit; wherein the target specification request includes the address information of the first source operand and the second source operand.

[0036] In this embodiment, the cache's request receiving circuit may receive a target reduction request sent by the target computing unit. The target computing unit can be any computing unit within the computing system. After independently completing local tensor operations, the target computing unit can synchronously initiate a reduction request to the cache via the on-chip bus. This target reduction request can be any reduction request issued by the target computing unit.

[0037] In this embodiment, when the request receiving circuit receives a target specification request, it can perform at least one of the following processes: caching, arbitration, and preprocessing on the received target specification request.

[0038] The target reduction request is a digital control instruction signal. The target reduction request can carry the address information of the first source operand and the second source operand. The address information refers to the address of the first source operand in the memory system, which may include the starting address, address range, etc., and is used to uniquely identify the storage location of the first source operand.

[0039] The first source operand can be stored in a cache or in memory. The second source operand can be stored in the target storage buffer corresponding to the target reduction request.

[0040] Furthermore, the target reduction request can also carry information such as the reduction operation type (summation, maximum, minimum, etc.) and the corresponding computation unit ID (identity). The computation unit ID can be the ID of the target computation unit that sent the target reduction request.

[0041] Step S220: If the first source operand does not hit the cache, allocate the corresponding target cache line for the target specification request.

[0042] A cache line is the smallest unit of data storage and allocation in the cache. The cache stores, retrieves, and replaces data in units of cache lines. Each cache line has a fixed size and is used to temporarily store the data required for operations or the results of operations.

[0043] In this embodiment, the target caching behavior is a dedicated cache line allocated specifically for the target reduction request. This cache line stores the data and / or reduction operation results required by the target reduction request, enabling fast temporary storage and access to the reduction operation results while ensuring data isolation and security between different reduction requests.

[0044] In this embodiment, the cache request receiving circuit can transmit the target reduction request to the hit detection circuit. The hit detection circuit can detect whether the first source operand hits the cache based on the address information of the first source operand. If the first source operand is stored in the cache, it is determined that the target reduction request hits the cache; if the first source operand is not stored in the cache, it is determined that the first source operand does not hit the cache.

[0045] Specifically, the cache control circuitry of the cache can allocate the corresponding target cache line for the target specification request.

[0046] In some embodiments, the hit detection circuit can also send the target specification request to a cached request scheduling circuit, which then performs unified scheduling, sorting, and path allocation on the received specification requests. The request scheduling circuit internally maintains at least one scheduling queue for temporarily storing and sorting the received specification requests, and then transmits the received specification requests sequentially to the request execution circuit for execution.

[0047] Step S230: The first step of reading the first source operand from memory and the second step of reading the second source operand from the target cache area corresponding to the target reduction request from the storage buffer and writing the second source operand into the target cache line are executed in parallel.

[0048] In this embodiment, the cached memory access circuit may interact with the memory and perform the first step, while the cached request execution circuit performs the second step.

[0049] In this embodiment, the storage buffer is an on-chip temporary storage resource located on the interaction path from the on-chip bus to the cache, used to temporarily store the second source operand of the reduction request. The target cache area is the storage location on the storage buffer where the second source operand of the target reduction request is temporarily stored.

[0050] In this embodiment, the start time of executing the first step and the start time of executing the second step can be the same or different, and no limitation is made here. The first step and the second step can be executed in parallel, which means that the second step can be executed at any time during the execution of the first step and before the first source operand is obtained.

[0051] Specifically, based on the address information of the first source operand, the exact storage location of the first source operand in memory can be quickly located. The cache's memory access circuit sends a read request to the memory. According to the read request, the memory reads the first source operand that matches the target reduction request from the corresponding storage location and writes it to the request execution circuit. After receiving the first source operand, the request execution circuit temporarily stores it in a temporary register inside the cache, waiting for the subsequent reduction operation result to be called.

[0052] The target cache area is a pre-defined fixed cache area specifically used to cache the second source operand of the target reduction request. Its storage address is fixed, enabling fast access without the need for complex address resolution and location operations. The request execution circuit can send a read request to the storage buffer, and the target storage buffer, based on the read request, reads the second source operand from the target cache area and transmits it to the request execution circuit.

[0053] Specifically, the time to read the second source operand from the target cache area of ​​the storage buffer is less than the time to read the first source operand from memory.

[0054] In some embodiments, the method further includes: notifying the target storage buffer to release the target cache region when the second source operand is written to the target cache line.

[0055] After the request execution circuit confirms that the second source operand has been successfully written to the target cache line, it sends a cache clearing notification to the storage buffer via a dedicated communication link between the cache and the storage buffer. This notification carries either the identifier of the second source operand or the identifier of the target cache region, indicating the specific cache region that the storage buffer needs to clear. Upon receiving the cache clearing notification, the storage buffer clears the target cache region, releasing the storage resources it occupies.

[0056] The cache clearing operation in this embodiment only targets the target cache area and will not affect the normal operation of other cache areas in the storage buffer, thus ensuring the overall stability of the storage buffer.

[0057] After the storage buffer releases the target cache area occupied by the second source operand, the target cache area can be used by new reduction requests to avoid affecting the execution of new reduction requests.

[0058] This embodiment enables efficient cyclical utilization of cache resources within the storage buffer, effectively reducing the problem of cache areas being occupied for extended periods by source operands corresponding to already processed reduction requests, preventing new reduction requests from timely acquiring cache space and forcing them to wait for cache release or occupy other cache areas. Especially in high-concurrency reduction request processing scenarios, multiple reduction requests can sequentially reuse the target cache area without needing to allocate dedicated cache space for each request, thus improving the cache utilization rate of the storage buffer and reducing cache resource waste. Furthermore, new reduction requests can directly use the idle target cache area to cache their corresponding second source operands without additional cache space application and allocation, shortening the preparation time for new reduction requests and indirectly increasing the throughput of the computing device in processing reduction requests. This ensures stable and efficient cache operation in high-concurrency scenarios, further enhancing the application advantages of computing devices in high-concurrency, low-latency computing scenarios.

[0059] In some embodiments, before performing the second step, the method further includes: identifying whether the target cache line is in an idle state; if the target cache line is not in an idle state, identifying whether the data in the target cache line is consistent with the corresponding data in memory; if consistent, clearing the target cache line to set the target cache line to an idle state.

[0060] In this embodiment, if no data is stored in the target cache line, it indicates that the target cache line is in an idle state; if data is stored in the target cache line, it indicates that the target cache line is not in an idle state.

[0061] In this embodiment, a target status bit can be set in the cache line to indicate whether the data in the corresponding cache line has been overwritten, so as to distinguish whether the cache line data is consistent with the memory data. When the cache line data is written, updated, or modified, the cache control circuit sets the target status bit of the cache line to valid, indicating that the cache line data is new data and has not yet been written back to memory. At this time, the cache line data is inconsistent with the memory data.

[0062] Furthermore, the cache control circuit can identify whether the data in the target cache line is consistent with the corresponding data in memory by reading whether the target status bit of the target cache line is valid. If the target status bit of the target cache line is valid, it indicates that the target cache line is dirty, that is, the data in the target cache line is inconsistent with the corresponding data in memory; if the target status bit of the target cache line is invalid, it indicates that the target cache line is not dirty, that is, the data in the target cache line is consistent with the corresponding data in memory.

[0063] In this embodiment, before writing the second source operand to the target cache line, it is first identified whether the cache line is in an idle state. For cache lines that are not in an idle state, it is further determined whether their data is consistent with the data in memory. Only when the data is consistent is the cache line cleared and set to an idle state before allocation. This ensures that the cache lines allocated to the target reduction request are all in a clean, idle state without residual data, avoiding contamination of the reduction operation by old data and ensuring the storage security and operational accuracy of the data required by the target reduction request. In some embodiments, if the target cache line is identified as being in an idle state, the second step can be executed directly.

[0064] In some embodiments, the method further includes: if the data in the target cache line is inconsistent with the corresponding data in memory, writing the data stored in the target cache line into memory and releasing the space of the target cache line to set the target cache line to an idle state.

[0065] If the data in the target cache line is inconsistent with the corresponding data in memory, the cache control circuit can initiate a write-back operation to write the latest data in the cache line back to the corresponding memory address. After the write-back operation is completed, an eviction operation is performed on the cache line to release the target cache line and mark the target status bit of the target cache line as invalid.

[0066] In this embodiment, the released cache lines can be allocated to the target reduction request after evict is completed. This allows for the dynamic allocation of available storage resources to the target reduction request when there are no free cache lines, ensuring that the source operands are loaded into the cache and the reduction operation results are executed normally. Moreover, this allocation method has a clear process and orderly timing, which not only ensures the efficient reuse of cache resources, but also provides a stable and reliable execution environment for the reduction operation, which is conducive to improving the throughput and stability of the reduction processing.

[0067] This embodiment demonstrates that when the data in the target cache line is inconsistent with the corresponding data in memory, writing the data from the cache line back to memory first, then releasing the cache line space and setting it to an idle state, effectively avoids the loss of dirty cache data, ensures data consistency between the cache and memory, and prevents system risks caused by data anomalies or loss. Simultaneously, while ensuring data security and reliability, setting the target cache line specification to an idle state provides clean, reliable, and usable cache resources for subsequent target specification requests, improving cache resource utilization. It also enhances the stability and security of system operation from the perspectives of data protection and cache management.

[0068] After the second source operand is written to the target cache line and before the target reduction request is completed, the cache control circuit prohibits the execution of the evict operation on the target cache line to prevent the source operand or intermediate result of the target reduction request from being evicted prematurely, thus ensuring the continuity and correctness of the reduction operation.

[0069] Step S240: Upon receiving the first source operand, perform a reduction operation on the first source operand and the second source operand that matches the target reduction request to obtain the reduction operation result of the target reduction request.

[0070] Once the memory access circuit completes reading the first source operand (i.e., the first step is complete), and the request execution circuit completes reading, writing, and sending the cache clear notification for the second source operand (i.e., the second step is complete), the request execution circuit, having completed both steps, reads the first source operand from the temporary register and the second source operand from the target cache line. Based on the reduction operation type that can be carried in the target reduction request, it performs a reduction operation on the two source operands that matches the target reduction request, and obtains the reduction operation result.

[0071] It should be noted that the first step can be completed first, the second step can be completed first, or the first and second steps can be completed simultaneously.

[0072] Step S250: Return the reduction operation result to the target computing unit.

[0073] In this embodiment, the result return circuit can be a cached circuit that returns the reduction operation result to the target computing unit that sent the target reduction request via the on-chip bus. The result return circuit can return the reduction operation result to the corresponding target computing unit based on the ID of the target computing unit that can be carried in the target reduction request.

[0074] Furthermore, the target reduction request identification information can be carried in the reduction operation result to ensure that the target computing unit can associate the reduction operation result with the corresponding target reduction request, which facilitates subsequent processing (such as result storage, feedback to external devices, etc.).

[0075] Through the embodiments of this disclosure, during the execution of a reduction request, the first step of reading the first source operand of the target reduction request from memory and the second step of reading the second source operand of the target reduction request from the target cache area of ​​the storage buffer and writing the second source operand into the target cache line are executed in parallel. This breaks the limitation of the prior art where the reading and writing of the two source operands of a reduction request are executed serially, realizing the synchronous advancement of the two core data preparation steps and shortening the overall data preparation time of the reduction request. Compared with the serial execution mode that waits for one step to complete before starting another step, this embodiment does not require additional waiting time. While the memory access circuit reads the first source operand, the request execution circuit simultaneously completes the reading and writing of the second source operand, providing a foundation for the rapid initiation of subsequent reduction operation results and improving the overall processing efficiency of the reduction request.

[0076] In some embodiments, the method further includes: when the target reduction request hits the cache, reading a second source operand from the target cache area, reading the data hit by the target reduction request from the cache, performing a reduction operation matching the target reduction request on the data hit by the target reduction request and the second source operand, obtaining the reduction operation result, and returning the reduction operation result to the target computing unit.

[0077] If the target reduction request hits the cache, it means that the cache stores the first source operands required by the first reduction request. Therefore, there is no need to read from memory. The data hit by the first reduction request can be directly reused from the cache, which can shorten the data preparation time of the reduction operation and improve the execution efficiency of the reduction request.

[0078] Furthermore, after returning the reduction operation result to the corresponding target computation unit, the cache can determine whether to retain the data hit by the first reduction request in the cache according to the caching strategy. If the target reduction request is a high-frequency access request, the data hit by the first reduction request is retained to facilitate quick hits by subsequent similar requests. If the target reduction request is a low-frequency access request, the data hit by the first reduction request in the cache is cleared to release cache resources, avoid cache resources being occupied by low-frequency data, and optimize cache utilization.

[0079] In some embodiments, the method further includes writing the reduction operation result to a target cache line.

[0080] The target cache behavior cache has a separate temporary storage unit allocated for the target specification request. This unit is used to store the calculation result corresponding to the target specification request, avoiding data confusion with the calculation results and source operands of other specification requests, while ensuring that the calculation result can be quickly called and read.

[0081] Specifically, the request execution circuit first sends a write request to the target cache line. This write request carries the identifier information of the target reduction request and the reduction operation result, used to confirm the correspondence between the written data and the target reduction request, ensuring write accuracy. Upon receiving the write request, the target cache line quickly completes the write operation of the reduction operation result and sends a write completion signal back to the request execution circuit. The write process can employ high-speed write logic, which significantly reduces write latency compared to writing to memory or a regular cache area, ensuring rapid storage of the operation results.

[0082] After receiving the write completion signal from the target cache line, the request execution circuit returns the reduction operation result to the target computing unit through the high-speed communication link between the cache and the target computing unit.

[0083] Based on this, the target computing unit can quickly read the reduction operation results from the target cache line as needed, thereby improving the efficiency of subsequent processing.

[0084] In this embodiment, after obtaining the reduction operation result, the reduction operation result is written to the target cache line allocated for the target reduction request. This enables the reduction operation result to be quickly stored and accessed nearby, avoiding subsequent repeated execution of the reduction operation and reducing data reading latency.

[0085] In some embodiments, the method further includes: executing other specification requests in parallel during the execution of the first step.

[0086] In this embodiment, the method for executing other specification requests can be referred to the method for executing the target specification request described in steps S210 to S250 of this embodiment, and will not be repeated here.

[0087] In this embodiment, the computing unit that issues other specification requests may be the same as or different from the target computing unit, and no limitation is made here.

[0088] In this embodiment, the cache may receive the target specification request first and then receive other specification requests, or it may receive other specification requests first and then receive the target specification request, or it may receive the target specification request and other specification requests simultaneously.

[0089] The request scheduling circuit can send the target specification request and other specification requests to the request execution circuit in sequence.

[0090] During the execution of the first step, other reduction requests are executed in parallel, which may include any of the following: During the execution of the first step, other reduction requests are received; during the execution of the first step, an idle cache line is allocated for other reduction requests; during the execution of the first step, a third step of reading the first source operand of other reduction requests from memory and a fourth step of reading the second source operand of other reduction requests from the storage buffer and writing it to the cache line allocated to other reduction requests are executed in parallel; during the execution of the first step, a third step of reading the first source operand of other reduction requests from memory is executed; during the execution of the first step, a reduction operation matching other reduction requests is performed on the first and second source operands of other reduction requests to obtain the reduction operation result of other reduction requests; during the execution of the first step, the reduction operation result of other reduction requests is returned to the corresponding computation unit.

[0091] In this embodiment, other reduction requests are processed in parallel during the execution of the first step, which can make full use of the idle cycles of the cache and execution units, avoid idle hardware resources, and improve the parallel processing capability of the cache and the throughput of reduction operations. At the same time, through the pipelined parallel execution of multiple reduction requests, the overall processing latency is effectively reduced, and the processing efficiency and resource utilization of the cache for a large number of reduction requests are improved without increasing additional hardware overhead.

[0092] In the example where the target specification request hits the cache, the lifecycle of the target specification request after entering the cache can be as follows: Figure 3 As shown, after the target reduction request is received, if the cache is hit, the reduction operation is performed directly, and then the corresponding reduction operation result is returned.

[0093] In the example where the target specification request does not hit the cache, the lifecycle of the target specification request after entering the cache can be as follows: Figure 4As shown, after a target reduction request is received, if there is a cache miss, the target cache line needs to be allocated to the target reduction request. If the target cache line is dirty, the data stored in the target cache line needs to be written to memory first, the space of the target cache line needs to be released, and then the target cache line needs to be allocated to the target reduction request. After allocating the target cache line to the target reduction request, the second source operand of the target reduction request can be read immediately from the target cache area of ​​the storage buffer, written to the target cache line, and the first source operand of the target reduction request can be read from memory. The process of reading data from memory can be very long and can be executed in parallel with other reduction requests. After the first source operand is returned, it is sent to the pipeline, the second source operand is retrieved from the target cache line, the reduction operation matching the target reduction request is performed, the result of the reduction operation is written to the target cache line, and then the reduction operation result is returned to the target computation unit.

[0094] exist Figure 3 and Figure 4 In the example shown, the horizontal arrow indicates the length of time the target specification request needs to occupy the target cache area of ​​the storage buffer.

[0095] In one embodiment, such as Figure 5 As shown, the request execution circuit can have three pipelines. Pipeline 1 handles general read, write, and reduce requests. Pipeline 2 writes the second source operand to the target cache line when the first source operand misses the cache. Pipeline 3 writes the data stored in the target cache line to memory. In actual implementation, these three pipelines can be merged into two data paths, or the same data path can be reused, but all three operating modes must be retained.

[0096] For a target reduction request where the first source operand misses the cache and is allocated a dirty target cache line, the target reduction request first derives an evict sub-request. This evict sub-request reads data from the target cache line through pipeline 3 and writes it to memory. Subsequently, the target reduction request derives a second sub-request to pipeline 2, reads the second source operand, and writes it directly to the target cache line. Simultaneously with generating the second sub-request, the request execution circuit can issue a miss request to the memory access circuit, which then retrieves the first source operand from memory based on the miss request. When the first source operand is returned, the target reduction request submits a third sub-request to pipeline 1, reads the second source operand stored in the target cache line, performs a reduction operation with the first source operand returned from memory, writes the reduction result to the target cache line, and returns the reduction result to the target computation unit through the result return circuit.

[0097] <Second Embodiment> This embodiment provides a cache, which can be the last-level cache of a computing device.

[0098] like Figure 6 As shown, the cache 112 includes a request receiving circuit 610, a cache control circuit 620, a memory access circuit 630, a request execution circuit 640, and a result return circuit 650.

[0099] The request receiving circuit 610 is configured to receive a target reduction request sent by the computing unit; wherein, the target reduction request includes address information of a first source operand and a second source operand; the cache control circuit 620 is configured to allocate a corresponding target cache line for the target reduction request if the target reduction request does not hit the cache; the memory access circuit 630 is configured to perform a first step of reading the first source operand matching the target reduction request from memory according to the address information of the target reduction request; the request execution circuit 640 is configured to perform a second step of reading the second source operand of the target reduction request from the storage buffer corresponding to the target reduction request and writing the second source operand into the target cache line when the memory access circuit 630 performs the first step; upon receiving the first source operand, a reduction operation is performed on the first source operand and the second source operand to obtain the reduction operation result of the target reduction request; the result return circuit 650 returns the reduction operation result to the computing unit.

[0100] <Third Embodiment> This disclosure also provides a computing device. Figure 7 This illustrates one hardware architecture of the computing device. The computing device can be any device capable of parallel computing, such as a GPU, GPGPU, NPU, or TPU.

[0101] like Figure 7 As shown, the computing device 110 includes a plurality of computing units 111, a cache 112, and an on-chip bus 113. The computing units 111 are coupled to the on-chip bus 113, and the on-chip bus 113 is coupled to the cache via at least one level of buffer. The computing units 111 can access the cache 112 through the on-chip bus.

[0102] In some examples, cache 112 can be a cache according to the second embodiment described above.

[0103] In other examples, cache 112 is configured to execute the specification request execution method, which may include the first embodiment described above.

[0104] In some examples, this cache is the last-level cache of the computing device with a large number of memory accesses.

[0105] <Fourth Embodiment> This disclosure also provides a computing system, such as Figure 8 As shown, the computing system 100 includes a control device 120 and a computing device 110. The computing device 110 may include a cache according to the third embodiment, or it may be a computing device according to the fourth embodiment, and it may also be configured to execute a specification request execution method according to the first embodiment. The control device 120 is used to control the computing device 110 to execute computing tasks, and the control device 120 is, for example, a CPU.

[0106] This computing device can be deployed in any type of electronic device, such as any type of terminal device, workstation, or server, without any limitation.

[0107] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and apparatuses according to various embodiments of this specification. In this regard, each block in a flowchart or block diagram may represent a module, unit, or part of a circuit. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented in hardware that performs the specified function or action, or in a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0109] Various embodiments of this specification have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for executing a specification request, characterized in that, Applied to caching, the method includes: Receive a target specification request sent by the target computing unit; wherein the target specification request includes the address information of the first source operand and the second source operand; If the first source operand does not hit the cache, a corresponding target cache line is allocated for the target reduction request; The first step of reading the first source operand from memory based on the address information and the second step of reading the second source operand from the target cache area corresponding to the target reduction request in the storage buffer and writing the second source operand into the target cache line are executed in parallel. Upon receiving the first source operand, perform a reduction operation on the first source operand and the second source operand that matches the target reduction request to obtain the reduction operation result of the target reduction request; The reduction operation result is returned to the target computing unit.

2. The method according to claim 1, characterized in that, The method further includes: Identify whether the target cache line is in an idle state; If the target cache line is not in an idle state, it is determined whether the data in the target cache line is consistent with the corresponding data in the memory. If the data in the target cache line is consistent with the corresponding data in the memory, the target cache line is cleared.

3. The method according to claim 2, characterized in that, The method further includes: If the data in the target cache line is inconsistent with the corresponding data in memory, the data stored in the target cache line is written to the memory and the space of the target cache line is released.

4. The method according to claim 1, characterized in that, The method further includes: If the first source operand hits the cache, the second source operand is read from the target cache region, and the first source operand is read from the cache; a reduction operation matching the target reduction request is performed on the first source operand and the second source operand to obtain the reduction operation result.

5. The method according to claim 1, characterized in that, The method further includes: Write the result of the reduction operation into the target cache line.

6. The method according to claim 1, characterized in that, The method further includes: When the second source operand is written to the target cache line, the storage buffer is notified to release the target cache region.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: During the execution of the second step, other specification requests are executed in parallel.

8. A cache, characterized in that, The cache includes: The request receiving circuit is configured to receive a target specification request sent by the computing unit; wherein the target specification request includes address information of a first source operand and a second source operand; The cache control circuit is configured to allocate a corresponding target cache line for the target specification request if the target specification request does not hit the cache. The memory access circuit is configured to perform a first step of reading a first source operand from memory that matches the target specification request based on the address information requested by the target specification request; The request execution circuit is configured to, when the memory access circuit executes the first step, execute the second step of reading the second source operand of the target reduction request from the storage buffer corresponding to the target reduction request and writing the second source operand into the target cache line; and, upon receiving the first source operand, perform a reduction operation on the first source operand and the second source operand to obtain the reduction operation result of the target reduction request. The result return circuit returns the reduction operation result to the computing unit.

9. The cache according to claim 8, characterized in that, The cache is a last-level cache.

10. A computing device, characterized in that, The computing device includes: Multiple computing units; A cache, wherein the cache is the cache of claim 8 or 9; or, the cache is configured to perform the method of any one of claims 1 to 7; and, An on-chip bus is provided, wherein the computing unit is coupled to the on-chip bus, and the on-chip bus is coupled to the cache.

11. A computing system, characterized in that, include: The control device and the computing device of claim 10, wherein the control device is used to control the computing device to perform computing tasks.

Images