caching and its operation methods

Abstract

This invention provides a cache and its operation method. The cache includes a reference table and a hit-miss checking unit. The hit-miss checking unit is coupled to the reference table. In response to one of a plurality of compute cores sending a replacement request to the cache, the hit-miss checking unit checks a corresponding reference bit in the reference table for the target cache line of the replacement request. In response to the corresponding reference bit in the reference table indicating that the target cache line is out of service, the hit-miss checking unit executes the replacement request. In response to the corresponding reference bit in the reference table indicating that the target cache line is busy, the hit-miss checking unit temporarily suspends the replacement request until the corresponding reference bit in the reference table indicates that the target cache line is out of service.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence (AI) chips, and particularly to a cache and its operation method. Background Technology

[0002] In the architectures of computing devices such as artificial intelligence (AI) chips, graphics processing units (GPUs), and general-purpose GPUs (GPGPUs), cache line replacement is crucial. Replacement refers to the system selecting a cache line that is not currently being used by any program when the cache is full and new data needs to be loaded. Ensuring that the selected cache line is not being used by any program is one of the many technical challenges in this field. Summary of the Invention

[0003] This invention relates to a cache and its operation method for securely executing replacement requests.

[0004] In an embodiment of the invention, the cache includes a referencing table and a hit-miss check unit. The hit-miss check unit is coupled to the referencing table. In response to one of a plurality of compute cores sending a replacement request to the cache, the hit-miss check unit checks a corresponding reference bit in the referencing table for the target cache line of the replacement request. In response to the corresponding reference bit in the referencing table indicating that the target cache line is retired, the hit-miss check unit executes the replacement request. In response to the corresponding reference bit in the referencing table indicating that the target cache line is busy, the hit-miss check unit temporarily suspends the replacement request until the corresponding reference bit in the referencing table indicates that the target cache line is retired.

[0005] In an embodiment of the present invention, the operation method includes: in response to one of a plurality of computing cores sending a replacement request to a cache, a cache hit-miss checking unit checks a corresponding reference bit in the cache's reference table corresponding to the target cache line of the replacement request; in response to the corresponding reference bit in the reference table indicating that the target cache line is out, the hit-miss checking unit executes the replacement request; and in response to the corresponding reference bit in the reference table indicating that the target cache line is busy, the hit-miss checking unit temporarily suspends the execution of the replacement request until the corresponding reference bit in the reference table indicates that the target cache line is out.

[0006] Based on the above, each cache line is configured with a dedicated reference bit. Each reference bit in the reference table indicates the reference status (or busy status) of a corresponding cache line. The reference status refers to whether the cache line is referenced (used) by any program. When the hit-miss check unit receives a replacement request, it checks the corresponding reference bit of the target cache line in the reference table. The corresponding reference bit ensures that the target cache line is either busy (referenced) or out of service (not referenced). Therefore, the cache can safely execute replacement requests. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of a circuit block of an artificial intelligence (AI) chip according to an embodiment of the present invention.

[0008] Figure 2 This is a flowchart illustrating an operation method of an artificial intelligence chip according to an embodiment of the present invention.

[0009] Figure 3 This is a schematic diagram of a reference table drawn according to an embodiment of the present invention.

[0010] Figure 4 This is a circuit block diagram of a management unit drawn according to an embodiment of the present invention.

[0011] Explanation of icon numbers

[0012] 100: AI chips

[0013] 110: Calculation kernel,

[0014] 120: Cache

[0015] 121: Hit-miss check unit,

[0016] 122: Tag array,

[0017] 123: Engine operation,

[0018] 124: Cache Array

[0019] 125: Reference Table,

[0020] 126: Management Unit

[0021] 130: Main memory,

[0022] 310_1: Group 1 referenced figure,

[0023] 310_2: Group 2 referenced figures,

[0024] 310_c: The cth group of referenced graphs,

[0025] 410: Exit engine.

[0026] 420: Arbitrator

[0027] ref_1: The first reference bit,

[0028] ref_2: the second reference bit,

[0029] ref_m: The m-th reference bit. Detailed Implementation

[0030] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same component symbols are used in the drawings and description to denote the same or similar parts.

[0031] The term "coupled (or connected)" as used throughout this specification (including the claims) may refer to any direct or indirect means of connection. For example, if the text describes a first device coupled (or connected) to a second device, it should be interpreted as the first device being directly connected to the second device, or the first device being indirectly connected to the second device through other devices or some means of connection. The terms "first," "second," etc., used throughout this specification (including the claims) are used to name components or distinguish different embodiments or scopes, and are not intended to limit the upper or lower limit of the number of components, nor to limit the order of components. Furthermore, wherever possible, components / components / steps using the same reference numerals in the drawings and embodiments represent the same or similar parts. Components / components / steps using the same reference numerals or the same terms in different embodiments may be referred to mutually in the relevant descriptions. It should be understood that the features of the following embodiments can be combined with each other. For example, the features of the second embodiment can be combined with the features of the first embodiment. Those skilled in the art can select appropriate combinations of features according to actual design requirements.

[0032] Computing devices such as artificial intelligence (AI) chips can provide enormous computing power. This immense computing power stems from the large number of internal hardware cores. An AI chip typically contains multiple programmable processors, such as a stream processor cluster (SPC). Each programmable processor typically contains multiple compute units (CUs), and each compute core typically contains multiple execution units (EUs). Execution cores may include at least one of the following: tensor cores (Tcores), integer cores (INTs), floating-point cores (FPs), and vector cores (Vcores). By organizing these compute cores through programming, AI chips can support general-purpose computing, scientific computing, and neural network computing. The compute cores of an AI chip typically access data in main memory through a cache, such as a last-level cache (LLC). The following examples illustrate cache implementation paradigms.

[0033] Figure 1 This is a schematic diagram of a circuit block of an AI chip 100 according to an embodiment of the present invention. Figure 1 In the illustrated embodiment, the AI ​​chip 100 includes multiple computing cores 110, a cache 120, and main memory 130. The number of computing cores 110 can be determined according to the actual design and application. A computing core 110 is also called a Compute Unit (CU). Each computing core 110 typically contains multiple Execution Units (EUs, or execution cores) and shared memory. Different execution cores within the same computing core can exchange data with each other through shared memory. The cache 120 is coupled between the computing cores 110 and the main memory 130. The cache 120 can be the last-level cache (LLC) of the AI ​​chip 100 or other caches. The computing cores 110 access data in the main memory 130 through the cache 120.

[0034] exist Figure 1In the illustrated embodiment, cache 120 includes a hit-miss check unit 121, a tag array 122, an operation engine 123, a cache line array 124, a referencing table 125, and a management unit 126. The hit-miss check unit 121 is coupled to the tag array 122, the operation engine 123, the referencing table 125, and the management unit 126, while the operation engine 123 is coupled to the cache line array 124. Depending on the design, in some embodiments, at least one of the hit-miss check unit 121, the operation engine 123, and the management unit 126 may be implemented as hardware circuitry. In other embodiments, at least one of the hit-miss check unit 121, the operation engine 123, and the management unit 126 may be implemented as a combination of hardware, firmware, and software (i.e., a program).

[0035] In terms of hardware, at least one of the aforementioned hit-miss check unit 121, operation engine 123, and management unit 126 can be implemented as logic circuitry on an integrated circuit. For example, the relevant functions of at least one of the hit-miss check unit 121, operation engine 123, and management unit 126 can be implemented in various logic blocks, modules, and circuits within one or more hardware controllers, microcontrollers, hardware processors, microprocessors, application-specific integrated circuits (ASICs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), central processing units (CPUs), or other processing units. The relevant functions of at least one of the hit-miss check unit 121, operation engine 123 and management unit 126 can be implemented as hardware circuits, such as various logic blocks, modules and circuits in integrated circuits, using hardware description languages ​​(such as Verilog HDL or VHDL) or other suitable programming languages.

[0036] In software or firmware form, the functions of at least one of the hit-miss checking unit 121, operating engine 123, and management unit 126 can be implemented as programming codes. For example, at least one of the hit-miss checking unit 121, operating engine 123, and management unit 126 can be implemented using general programming languages ​​(such as C, C++, or assembly language) or other suitable programming languages. The programming codes can be recorded / stored in a non-transitory machine-readable storage medium. In some embodiments, the non-transitory machine-readable storage medium includes, for example, semiconductor memory and / or a storage device. An electronic device (e.g., a CPU, hardware controller, microcontroller, hardware processor, or microprocessor) can read and execute the programming codes from the non-transitory machine-readable storage medium to implement the functions of at least one of the hit-miss checking unit 121, operating engine 123, and management unit 126.

[0037] Tag array 122 includes multiple tag sets, and each tag set includes multiple tags. Cache line array 124 includes multiple cache lines. The multiple cache lines of cache line array 124 correspond one-to-one with the multiple tags of tag array 122. Each tag is used to store the corresponding cache line information. Hit-miss checking unit 121 retrieves the corresponding tag set from tag array 122 based on the group address, and then compares the tag address field of each tag in the corresponding tag set with the tag address carried in the access request. If the tag address field content of a tag matches the tag address carried in the access request, hit-miss checking unit 121 determines "hit". If the tag address field content of any tag does not match the tag address carried in the access request, hit-miss checking unit 121 determines "miss".

[0038] Operation engine 123 is coupled to hit-miss check unit 121 and cache line array 124. In response to one of the plurality of compute cores 110 sending an access request to cache 120, hit-miss check unit 121 checks whether the access request is a hit. In response to hit-miss check unit 121 determining that the access request is a hit, operation engine 123 accesses the cache line corresponding to the access request among the plurality of cache lines in cache line array 124. In response to hit-miss check unit 121 determining that the access request is a miss, operation engine 123 accesses main memory 130.

[0039] Figure 2 This is a flowchart illustrating an operation method of an artificial intelligence chip according to an embodiment of the present invention. Please refer to... Figure 1 and Figure 2 In response to one of the multiple computing cores 110 sending a replacement request to the cache 120, the hit-miss check unit 121 checks the corresponding reference bit in the reference table 125 for the target cache line of the replacement request (step S210). The specific structure of the reference table 125 can be determined according to the actual design and application.

[0040] Figure 3 This is a schematic diagram illustrated according to an embodiment of the present invention, referencing Table 125. Figure 3 The referenced Table 125 shown can be used as... Figure 1 The example shown is one of many implementation examples referenced in Table 125. In Figure 3 In the illustrated embodiment, reference table 125 includes multiple set referring maps, such as reference map 310_1, reference map 310_2, ..., reference map 310_c. Reference maps 310_1 to 310_c in reference table 125 correspond one-to-one with multiple cache line groups of cache line array 124. Each of the multiple cache line groups includes multiple cache lines. Each reference map 310_1 to 310_c includes multiple reference bits, such as reference bit ref_1, reference bit ref_2, ..., reference bit ref_m. Reference bits ref_1 to ref_m in reference maps 310_1 to 310_c correspond one-to-one with different cache lines of cache line array 124.

[0041] In reference table 125, the first reference bit ref_1 to the m-th reference bit ref_m of reference diagrams 310_1 to 310_c (group 1) are used to indicate the reference status (or busy status) of different corresponding cache lines. The reference status refers to whether a cache line is referenced (used) by any program. For example (but not limited to this), suppose a cache line of cache line array 124 corresponds to the first reference bit ref_1 of reference diagram 310_1. When a cache line is referenced (used) by one or more programs, the first reference bit ref_1 of reference diagram 310_1 is set to logical "truth", such as the logical value "1". Conversely, when a cache line is not referenced (used) by any program, the first reference bit ref_1 of reference diagram 310_1 is reset to logical "false", such as the logical value "0".

[0042] Please refer to Figure 1 , Figure 2 and Figure 3 In response to the corresponding reference bit in reference table 125 indicating that the target cache line is retired (i.e., not referenced), the hit-miss check unit 121 executes the replacement request (step S220). In response to the corresponding reference bit in reference table 125 indicating that the target cache line is busy (referenced), the hit-miss check unit 121 temporarily suspends the replacement request until the corresponding reference bit in reference table 125 indicates that the target cache line is retired (step S230).

[0043] Management unit 126 manages reference table 125. Management unit 126 is coupled to hit-miss checking unit 121 and operation engine 123. In response to hit-miss checking unit 121 executing a replacement request, management unit 126 sets the corresponding reference bit for the target cache line of the replacement request in reference table 125. In response to one of the plurality of compute cores 110 sending an access request to the cache, management unit 126 sets the reference bit for the target cache line of the access request in reference table 125.

[0044] In response to the operation engine 123 completing the current access request, the management unit 126 checks whether the target cache line of the access requests to be executed by the operation engine 123 is the same as the target cache line of the current access request. If the target cache line of any of the access requests to be executed by the operation engine 123 is the same as the target cache line of the current access request, the management unit 126 maintains the reference bit corresponding to the target cache line of the access request in the reference table 125 in the set state. If the target cache lines of all the access requests to be executed by the operation engine 123 are different from the target cache line of the current access request, the management unit 126 resets the reference bit corresponding to the target cache line of the access request in the reference table 125.

[0045] In summary, each cache line of cache line array 124 is configured with a dedicated reference bit. Each reference bit in reference table 125 is used to indicate the reference status (or busy status) of a corresponding cache line. The reference status refers to whether the cache line is referenced (used) by any program. When the hit-miss check unit 121 receives a replacement request, the hit-miss check unit 121 checks the corresponding reference bit of the target cache line of the replacement request described in reference table 125. The corresponding reference bit in reference table 125 can guarantee that the target cache line is busy (referenced) or out of service (not referenced). Therefore, cache 120 can safely execute the replacement request.

[0046] Figure 4 This is a circuit block diagram of the management unit 126 according to an embodiment of the present invention. Figure 4 The management unit 126 shown can be used as Figure 1 This is one of many implementation examples of the management unit 126 shown. Figure 4 The hit-miss check unit 121, operation engine 123, reference table 125, and management unit 126 shown can be referenced. Figures 1 to 3 The relevant explanations. In Figure 4 In the illustrated embodiment, management unit 126 includes an exit engine 410 and an arbitrator 420. The exit engine 410 is coupled to the operation engine 123. The arbitrator 420 manages the reference table 125. The arbitrator 420 is coupled to the exit engine 410 and a hit-miss check unit 121. In response to the hit-miss check unit 121 executing a replacement request, the arbitrator 420 sets the corresponding reference bit in the reference table 125 corresponding to the target cache line of the replacement request. In response to one of the compute cores 110 sending an access request to the cache 120, the arbitrator 420 sets the reference bit in the reference table 125 corresponding to the target cache line of the access request.

[0047] In response to the operation engine 123 completing the current access request, the exit engine 410 checks whether the target cache line of the pending access requests in the operation engine 123 is the same as the target cache line of the current access request. If the check result of the exit engine 410 indicates that the target cache line of any pending access request in the operation engine 123 is the same as the target cache line of the current access request, the arbitrator 420 maintains the set state of the reference bit corresponding to the target cache line of the access request in the reference table 125. If the check result of the exit engine 410 indicates that the target cache lines of all pending access requests in the operation engine 123 are different from the target cache line of the current access request, the arbitrator 420 resets the reference bit corresponding to the target cache line of the access request in the reference table 125.

[0048] Sometimes, the set command issued by the hit-miss check unit 121 to the arbitrator 420 and the reset command issued by the exit engine 410 to the arbitrator 420 may arrive at the arbitrator 420 simultaneously. In this case, the arbitration strategy of the arbitrator 420 includes that the set command issued by the hit-miss check unit 121 to the arbitrator 420 takes precedence over the reset command issued by the exit engine 410 to the arbitrator 420.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cache, characterized by, The cache comprises: a reference table; and a hit-miss check unit coupled to the reference table, wherein in response to one of the plurality of compute cores sending a replacement request to the cache, the hit-miss check unit checks a corresponding reference bit in the reference table corresponding to a target cache line of the replacement request; in response to the corresponding reference bit in the reference table corresponding to the target cache line indicating that the target cache line is evicted, the hit-miss check unit performs the replacement request; and in response to the corresponding reference bit in the reference table corresponding to the target cache line indicating that the target cache line is busy, the hit-miss check unit temporarily does not perform the replacement request until the corresponding reference bit in the reference table corresponding to the target cache line of the replacement request indicates that the target cache line is evicted.

2. The cache of claim 1, wherein, The cache further comprises: a cache line array comprising a plurality of cache line groups, wherein each of the plurality of cache line groups comprises a plurality of cache lines, the plurality of cache lines of the cache line array one-to-one corresponding to different reference bits of the reference table; and an operation engine coupled to the hit-miss check unit and the cache line array, wherein in response to one of the plurality of compute cores sending an access request to the cache, the hit-miss check unit checks whether the access request is a hit; in response to the hit-miss check unit determining that the access request is a hit, the operation engine accesses a cache line corresponding to the access request among the plurality of cache lines of the cache line array; and in response to the hit-miss check unit determining that the access request is a miss, the operation engine accesses a main memory.

3. The cache of claim 2, wherein, The reference table comprises a plurality of group reference maps, the plurality of group reference maps one-to-one corresponding to the plurality of cache line groups of the cache line array, each of the plurality of group reference maps comprising a plurality of reference bits, the plurality of reference bits one-to-one corresponding to the plurality of cache lines of the cache line array.

4. The cache of claim 3, wherein, The cache further comprises: a management unit to manage the reference table, wherein the management unit is coupled to the hit-miss check unit and the operation engine, and in response to the hit-miss check unit performing the replacement request, the management unit sets the corresponding reference bit in the reference table corresponding to the target cache line of the replacement request.

5. The cache of claim 4, wherein in response to one of the plurality of compute cores sending the access request to the cache, the management unit sets a reference bit in the reference table corresponding to a target cache line of the access request.

6. The cache of claim 4, wherein in response to the operation engine completing a current access request, the management unit checks whether a target cache line of a pending access request of the operation engine is the same as a target cache line of the current access request; and in response to the target cache line of the pending access request of the operation engine being the same as the target cache line of the current access request, the management unit sets the reference bit in the reference table corresponding to the target cache line of the pending access request. in response to any one of the pending access requests of the operation engine having the same target cache line as the target cache line of the current access request, the management unit maintaining a set state of a reference bit in the reference table corresponding to the target cache line of the access request; and in response to all of the pending access requests of the operation engine having different target cache lines than the target cache line of the current access request, the management unit resetting the reference bit in the reference table corresponding to the target cache line of the access request.

7. The cache of claim 4, wherein, The management unit comprises: an exit engine coupled to the operation engine; and an arbiter to manage the reference table, wherein the arbiter is coupled to the exit engine and the hit-miss check unit, wherein, in response to the hit-miss check unit executing the replacement request, the arbiter setting the corresponding reference bit in the reference table corresponding to the target cache line of the replacement request.

8. The cache of claim 7, wherein, in response to one of the plurality of compute cores sending the access request to the cache, the arbiter setting the reference bit in the reference table corresponding to the target cache line of the access request.

9. The cache of claim 7, wherein, in response to the operation engine completing a current access request, the exit engine checking whether the target cache line of any one of the pending access requests of the operation engine is the same as the target cache line of the current access request; in response to the exit engine checking result indicating that the target cache line of any one of the pending access requests of the operation engine is the same as the target cache line of the current access request, the arbiter maintaining the set state of the reference bit in the reference table corresponding to the target cache line of the access request; and in response to the exit engine checking result indicating that the target cache line of all of the pending access requests of the operation engine are different from the target cache line of the current access request, the arbiter resetting the reference bit in the reference table corresponding to the target cache line of the access request.

10. The cache of claim 7, wherein, The hit-miss check unit has a higher priority than the exit engine to issue a set command to the arbiter.

11. A method of operating a cache, the method comprising: The operation method comprises: in response to one of the plurality of compute cores sending the access request to the cache, the hit-miss check unit checking a corresponding reference bit in a reference table of the cache corresponding to a target cache line of the access request, wherein the hit-miss check unit is coupled to the reference table; in response to the corresponding reference bit in the reference table corresponding to the target cache line indicating that the target cache line is an exit, the hit-miss check unit executing the replacement request; and in response to the corresponding reference bit in the reference table corresponding to the target cache line indicating that the target cache line is an exit, the hit-miss check unit executing the replacement request; and in response to the corresponding reference bit in the reference table corresponding to the target cache line of the replacement request indicating that the target cache line is busy, temporarily not performing the replacement request by the hit-miss check unit until the corresponding reference bit in the reference table corresponding to the target cache line of the replacement request indicates that the target cache line is retired.

12. The method of claim 11, wherein, The cache line array of the cache comprises a plurality of cache line groups, each of the plurality of cache line groups comprises a plurality of cache lines, the plurality of cache lines of the cache line array correspond to different reference bits of the reference table one-to-one, and the operation method further comprises: in response to one of the plurality of computing cores sending an access request to the cache, checking by the hit-miss check unit whether the access request is a hit; in response to the hit-miss check unit determining that the access request is a hit, accessing by the operation engine of the cache a cache line corresponding to the access request in the plurality of cache lines of the cache line array, wherein the operation engine is coupled to the hit-miss check unit and the cache line array; and in response to the hit-miss check unit determining that the access request is a miss, accessing by the operation engine a main memory.

13. The method of operation of claim 12, wherein, The reference table comprises a plurality of group reference maps, the plurality of group reference maps correspond to the plurality of cache line groups of the cache line array one-to-one, each of the plurality of group reference maps comprises a plurality of reference bits, and the plurality of reference bits correspond to the plurality of cache lines of the cache line array one-to-one.

14. The method of claim 13, wherein, The operation method further comprises: managing by a management unit of the cache the reference table, wherein the management unit is coupled to the hit-miss check unit and the operation engine; and in response to the hit-miss check unit performing the replacement request, setting by the management unit the corresponding reference bit in the reference table corresponding to the target cache line of the replacement request.

15. The method of operation of claim 14, wherein, The operation method further comprises: in response to one of the plurality of computing cores sending the access request to the cache, setting by the management unit a reference bit in the reference table corresponding to the target cache line of the access request.

16. The method of claim 14, wherein, The operation method further comprises: in response to the operation engine completing a current access request, checking by the management unit whether the target cache line of a to-be-executed access request of the operation engine is the same as the target cache line of the current access request; in response to the target cache line of any one to-be-executed access request of the operation engine being the same as the target cache line of the current access request, maintaining by the management unit the set state of the reference bit in the reference table corresponding to the target cache line of the access request; and in response to the target cache lines of all to-be-executed access requests of the operation engine being different from the target cache line of the current access request, resetting by the management unit the reference bit in the reference table corresponding to the target cache line of the access request.

17. The method of claim 14, wherein, The operation method further comprises: the reference table is managed by an arbiter of the management unit, wherein the arbiter is coupled to an exit engine of the management unit and the hit-miss check unit, and the exit engine is coupled to the operation engine; and in response to the hit-miss check unit performing the replacement request, setting, by the arbiter, the corresponding reference bit in the reference table corresponding to the target cache line of the replacement request.

18. The method of operation of claim 17, wherein, The operation method further comprises: in response to one of the plurality of compute cores sending the access request to the cache, setting, by the arbiter, a reference bit in the reference table corresponding to the target cache line of the access request.

19. The method of claim 17, wherein, The operation method further comprises: in response to the operation engine completing a current access request, checking, by the exit engine, whether a target cache line of a pending access request of the operation engine is the same as a target cache line of the current access request; in response to a result of the checking by the exit engine indicating that the target cache line of any one of the pending access requests of the operation engine is the same as the target cache line of the current access request, maintaining, by the arbiter, a set state of a reference bit in the reference table corresponding to the target cache line of the access request; and in response to the result of the checking by the exit engine indicating that the target cache line of all of the pending access requests of the operation engine is different from the target cache line of the current access request, resetting, by the arbiter, the reference bit in the reference table corresponding to the target cache line of the access request.

20. The operating method according to claim 17, characterized in that, The hit-miss check unit has a higher priority than the exit engine in issuing a set command to the arbiter.

Images