Cache access device, coprocessor, heterogeneous computing system, and cache access method

Abstract

The embodiment of the invention provides a cache access device, a coprocessor, a heterogeneous computing system and a cache access method. The cache access device comprises: a first cache management unit, which queries a first cache management table to obtain a first cache state according to a cache request initiated by a coprocessor, the first cache management table indicating whether each memory address range of a memory space has the first cache state of cache data in a first cache space in a main processor; if the first cache state indicates that no cache data exists in the first cache space, the second cache management unit searches a second cache management table according to a cache request initiated by the coprocessor to obtain a second cache state, the second cache management table indicates whether different memory addresses have a second cache state of cache data in a plurality of second cache spaces in the coprocessor or not; and the cache access execution unit is used for executing the access of the cache request according to the second cache state.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a cache access device, a coprocessor, a heterogeneous computing system, and a cache access method. Background Art

[0002] With the rapid development of artificial intelligence, large language models such as CHAT GPT and LLAMA have an increasing demand for computing power on the underlying chips. For the current mainstream GPU / AI chips, the access bandwidth of DDR (Double Data Rate DRAM) / HBM (High Bandwidth Memory) is the performance design bottleneck of the entire chip. At the same time, with the development of GPU / AI chips, the demand for data interaction between main processors such as CPUs and coprocessors such as GPUs is also increasing.

[0003] When the amount of data exchanged between the main processor and the coprocessor is relatively small, there is a shared memory space between the main processor and the coprocessor. The main processor and the coprocessor can access the shared memory space without going through their own caches (i.e., non-cache access) to avoid cache data consistency issues.

[0004] If the non-cache access between the main processor and the coprocessor is changed to cache access, the efficiency of data interaction between the main processor and the coprocessor can be improved, but it will also cause the problem of cache data consistency. Summary of the invention

[0005] In view of this, embodiments of the present invention provide a cache access device, a coprocessor, a heterogeneous computing system, and a cache access method to solve the above problems.

[0006] According to a first aspect of an embodiment of the present invention, a cache access device is provided, comprising: a first cache management unit, querying a first cache management table according to a target memory address in a cache request initiated by a coprocessor, to obtain a first cache state corresponding to the target memory address, wherein the first cache management table indicates a first cache state of whether there is cache data in a first cache space in the main processor for each memory address range of the memory space; a second cache management unit, querying a second cache management table according to a cache request initiated by the coprocessor, to obtain a second cache state of a cache line corresponding to the target memory address and the coprocessor, if the first cache state indicates that there is no cache data in the first cache space, wherein the second cache management table indicates a second cache state of whether there is cache data in multiple second cache spaces in the coprocessor at different memory addresses; and a cache access execution unit, executing access to the cache request according to the second cache state.

[0007] In another implementation of the present invention, the cache access device also includes a queue management unit, which manages a cache request queue for a target memory address, and determines a cache request initiated by a processor whose cache status corresponding to the target memory address indicates the presence of the cache data as being prioritized for dequeuing from the cache request queue.

[0008] In another implementation of the present invention, the queue management unit is specifically used to: if the first cache status indicates that there is cache data in the first cache space, then the cache request initiated by the main processor in the cache request queue is determined to be prioritized for dequeue; if the second cache status indicates that there is cache data in the corresponding second cache space, then the cache request initiated by the coprocessor in the cache request queue is determined to be prioritized for dequeue.

[0009] In another implementation of the present invention, the first cache management table includes multiple bits, each bit indicates a different memory address range, and the bit value of each bit indicates a first cache state of the memory address range corresponding to the bit.

[0010] In another implementation of the present invention, the second cache management table includes multiple cache lines, and the cache attributes of each cache line include at least the memory address range corresponding to the cache data of the cache line, and whether the cache line has a second cache state of the cache data in the corresponding second cache space.

[0011] In another implementation of the present invention, the second cache state includes a hit state of the cache line and an available state of the cache data of the cache line. The cache access execution unit is specifically used to: if the hit state of the cache line indicates a hit, and the available state of the cache data of the cache line indicates available, then execute the access of the cache request for the cache line; if the hit state of the cache line indicates a miss, then execute the access of the cache request for the target memory address; if the hit state of the cache line indicates a hit, and the available state of the cache data of the cache line indicates unavailable, then suspend the execution of the access of the cache request until the available state of the cache data of the cache line indicates available.

[0012] In another implementation of the present invention, the first cache management unit is also used to: if the first cache state indicates that there is cache data in the first cache space, send a first refresh request to the main processor and change the first cache state of the memory address range where the target memory address is located, wherein the first refresh request is used to refresh the cache data in the first cache space to the memory address range where the target memory address is located.

[0013] In another implementation of the present invention, the first cache management unit is also used to: query the first cache management table according to the target memory address of the cache request initiated by the main processor to obtain a first cache state corresponding to the target memory address; if the first cache state indicates that there is cache data in the first cache space, the cache access execution unit executes the cache request initiated by the main processor for the data of the target memory address in the memory space.

[0014] In another implementation of the present invention, the second cache management unit is also used to: if the first cache status indicates that there is no cache data in the first cache space, then determine whether the second cache status indicates that there is cache data in the corresponding second cache space; if so, the cache access execution unit executes the cache request initiated by the main processor for the second cache space of the coprocessor.

[0015] According to a second aspect of an embodiment of the present invention, a coprocessor is provided, comprising: a plurality of processing units, each processing unit comprising: a computing unit, initiating a cache request; and a cache access device according to the first aspect.

[0016] According to a third aspect of an embodiment of the present invention, a heterogeneous computing system is provided, comprising: a main processor; a coprocessor according to the second aspect; and an interconnect bus connected between the main processor and the coprocessor.

[0017] According to a fourth aspect of an embodiment of the present invention, a cache access method is provided, comprising: querying a first cache management table according to a target memory address in a cache request initiated by a coprocessor to obtain a first cache state corresponding to the target memory address, wherein the first cache management table indicates a first cache state of whether there is cache data in the first cache space in the main processor for each memory address range of the memory space; if the first cache state indicates that there is no cache data in the first cache space, then the second cache management unit searches a second cache management table according to the cache request initiated by the coprocessor to obtain a second cache state of the cache line corresponding to the target memory address and the coprocessor, wherein the second cache management table indicates a second cache state of whether there is cache data in multiple second cache spaces in the coprocessor at different memory addresses; and executing access to the cache request according to the second cache state.

[0018] In the scheme of the embodiment of the present invention, the first cache management unit implements consistency management of the first cache space of the main processor and multiple second cache spaces between the coprocessor through the first cache management table, and decouples the first cache space from the second cache space. In addition, the second cache management unit implements consistency management of the multiple second cache spaces of the coprocessor through the second cache management table, and realizes decoupling between the multiple second cache spaces. Therefore, in a heterogeneous computing system including a main processor and a coprocessor, cache data consistency is improved while maintaining data interaction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0020] Figure 1 A schematic architectural diagram of a data access system applicable to some embodiments of the present invention.

[0021] Figure 2 A structural block diagram of a cache access device according to some embodiments of the present invention.

[0022] Figure 3A for Figure 2 A structural block diagram of a heterogeneous computing system to which the embodiments are applicable.

[0023] Figure 3B for Figure 2 A structural block diagram of a further example of an embodiment of the present invention.

[0024] Figure 4A for Figure 2 A schematic diagram of a memory space applicable to an embodiment of the present invention.

[0025] Figure 4B for Figure 2 A schematic diagram of a first cache management table of an embodiment of the present invention.

[0026] Figure 5A for Figure 2 Schematic diagram of a queue management mechanism for a cache request queue of some examples of embodiments.

[0027] Figure 5B for Figure 2 Schematic diagram of the queue management mechanism of the cache request queue of some other examples of the embodiments.

[0028] Figure 5C for Figure 2 Schematic diagram of the queue management mechanism of the cache request queue of some other examples of the embodiments.

[0029] Figure 5D for Figure 2 A schematic diagram of cache attributes of cache lines of a second cache management table of an embodiment of the present invention.

[0030] Figure 6 The present invention is a flowchart of the steps of a cache access method according to some other embodiments of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0032] The specific implementation of the embodiment of the present invention is further described below in conjunction with the accompanying drawings of the embodiment of the present invention.

[0033] Figure 1 A schematic architecture diagram of a data access system to which some embodiments of the present invention are applicable is shown. Figure 1 The data access system includes heterogeneous computing systems and memories. Figure 1 As shown, the heterogeneous computing system includes a main processor such as a CPU and a coprocessor such as a GPU, and the memory may include DDR (Double Data Rate DRAM) and / or HBM (High Bandwidth Memory).

[0034] In addition, the main processor and the coprocessor may be connected by an interconnect bus such as CXL, PCIE, etc. The main processor may include at least one level of cache, such as a last level cache (LLC). The coprocessor may include multiple processing units, such as Figure 1 Each processing unit may include its own computing core and at least one level of cache, and at least one level of cache has Figure 1 In addition, different processing units can be interconnected by a network on chip (NoC). Each processing unit can also be connected to a bus between the coprocessor and the memory through a memory access interface to access data in the memory space according to the memory address.

[0035] When cache access is used between the main processor and the coprocessor, the efficiency of data exchange between the main processor and the coprocessor is improved, but it also causes the problem of cache data consistency. For this reason, a series of schemes in the embodiments of the present invention can improve the cache data consistency between the main processor and the coprocessor while maintaining the data exchange efficiency.

[0036] Figure 2 The cache access apparatus according to some embodiments of the present invention is shown. Figure 2 The cache access device 200 includes a first cache management unit 210 , a second cache management unit 220 and a cache access execution unit 230 .

[0037] Specifically, the first cache management unit 210 queries the first cache management table according to the target memory address in the cache request initiated by the coprocessor to obtain the first cache state corresponding to the target memory address. The first cache management table indicates whether there is a first cache state of cached data in the first cache space of the main processor for each memory address range of the memory space.

[0038] In addition, if the first cache state indicates that there is no cache data in the first cache space, the second cache management unit 220 searches the second cache management table according to the cache request initiated by the coprocessor to obtain the second cache state of the cache line corresponding to the target memory address and the coprocessor. The second cache management table indicates the second cache state of whether there is cache data in multiple second cache spaces in the coprocessor at different memory addresses.

[0039] The cache access execution unit 230 executes the access of the cache request according to the second cache state.

[0040] It should be understood that the first cache space of the main processor may be the cache space of any level cache in the main processor, and the second cache space of the coprocessor may be the cache space of any level cache in the processing unit of the coprocessor.

[0041] In the scheme of the embodiment of the present invention, the first cache management unit implements consistency management of the first cache space of the main processor and multiple second cache spaces between the coprocessor through the first cache management table, and decouples the first cache space from the second cache space. In addition, the second cache management unit implements consistency management of the multiple second cache spaces of the coprocessor through the second cache management table, and realizes decoupling between the multiple second cache spaces. Therefore, in a heterogeneous computing system including a main processor and a coprocessor, cache data consistency is improved while maintaining data interaction efficiency.

[0042] In some examples, such as Figure 3A As shown, the main processor and the coprocessor form a heterogeneous computing system, which can be applied to Figure 1 The cache access device 200 may be provided in at least one of the plurality of processing units in the coprocessor. Figure 3A In the example, the multiple processing units are implemented as multiple dies, for example, Die0 and Die1. In addition, the local cache access device 200 can communicate with the computing core of the local processing unit, for example, receiving a cache request initiated by the computing core. The local cache access device 200 also communicates with the cache access device 200 of the remote processing unit to forward the cache request initiated by the remote computing core through the remote cache access device 200. In addition, each cache access device 200 can also be connected to the bus between the coprocessor and the memory through a local memory access interface to access data in the memory space according to the memory address. In addition, each cache access device 200 can also access the main processor through an interconnection bus such as CXL, PCIE, etc., for example, receiving a cache request initiated by the main processor. It should be understood that the main processor and the coprocessor access each other through the interconnection bus.

[0043] Furthermore, if Figure 3BAs shown, the cache access device 200 also includes a queue management unit 240, which divides the acquired cache requests into different cache request queues, and each cache request in each cache request queue is for the same memory address. Specifically, the queue management unit manages the cache request queue for the target memory address, and determines the cache request initiated by the processor whose cache status corresponding to the target memory address indicates the existence of cache data as a priority to be dequeued from the cache request queue, thereby improving the reliability and efficiency of cache access. For another example, the dequeue order of each cache request that is not dequeued with priority can be consistent with the enqueue order, thereby improving the management efficiency of the cache request queue.

[0044] When different cache requests are sent to the cache access device, the queue management unit first determines the execution order of multiple requests that arrive at the same time through arbitration. Once the execution order is confirmed, the cache access needs to first query the first cache management table, then query the second cache management table, and then forward the cache request to the corresponding execution device according to the query result of the second cache management table.

[0045] Furthermore, the first cache management unit 210 includes a first storage unit 211 for temporarily storing the first cache management table, and a first query unit 212. The second cache management unit 220 includes a second storage unit 221 for temporarily storing the second cache management table, and a second query unit 222.

[0046] It should be understood that the cache access execution unit 230 can be connected to the memory access interface, the computing core, and the second cache space in the processing unit to access the memory space via the memory access interface, directly access the second cache space, and return a response to the cache request to the computing core.

[0047] In other examples, such as Figure 4A As shown, the interconnection bus between the main processor and the coprocessor divides the memory space of the coprocessor in the memory 300 into a HDM (host managed device memory) space and a non-HDM space. The CPU only performs cache access to the HDM space, and only performs non-cache access (none cacheable load / store) to the non-HDM space.

[0048] In the case of non-cache access, the main processor writes the data required for the operation of the coprocessor into the non-HDM space of the coprocessor. Specifically, the coprocessor reads the data from the non-HDM space into the cache of the coprocessor and performs operations. Then, the coprocessor writes the operation results from the cache back to the non-HDM space. Then, the main processor retrieves the operation results of the coprocessor from the non-HDM space. Among them, the main processor is in a multi-computing core working mode, and multiple computing cores may read the data stored in the non-HDM space multiple times.

[0049] In the case of cache access, the first cache management table indicates whether each memory address range of the memory space has a first cache state of cached data in the first cache space of the main processor. Figure 4B As shown, the first cache management table may be an HDM table. For example, the amount of data managed by the HDM table includes the product of the number of bits and the size represented by each bit. In other words, the first cache management table includes a plurality of bits, each bit indicates a different memory address range, and the bit value of each bit indicates the first cache state of the memory address range corresponding to the bit.

[0050] The cache request initiated by the main processor includes the target memory address. When the target memory address has cache data in the first cache space, the cache request directly accesses the first cache space. In this case, the cache request initiated by the main processor is implemented using the memory instruction in the main processor. When the target memory address does not have cache data in the first cache space, the cache request is sent to the coprocessor using the interconnection bus between the main processor and the coprocessor to join the cache request queue. For example, the cache request is encapsulated using the bus instruction of the interconnection bus, and the bus instruction includes a field for the main processor's identification and a field for the target memory address.

[0051] like Figure 4B As shown, the number of bits of the HDM table is 65535, each bit is represented as a grid, and each bit indicates a different memory address range. For example, each bit can represent a size of 4KB / 1KB / 256B, etc., depending on the total HDM space size required by the software.

[0052] In addition, the smaller the memory address range represented by each bit, the less data is refreshed by each first refresh request of the main processor, and the better the consistency of the cache data. The larger the memory address range represented by each bit, the larger the total memory space that can be represented by the HDM table. In addition, the memory address ranges represented by each bit do not overlap with each other. In some examples, the size of the memory address range represented by each bit is the same.

[0053] In some examples, when the main processor performs a cache access, if the memory address of the cache access falls within the corresponding grid, the bit of the grid of the memory address is flipped to the H bias state, which indicates that the cache data of the memory address exists in the first cache space of the main processor. When the main processor abandons the cache data of the memory address, the table will be flipped to the D bias state.

[0054] When the coprocessor performs a cache access, if the memory address of the cache access falls within the corresponding grid, the bit of the grid corresponding to the memory address is flipped to the D bias state (i.e., the bit value of the bit is flipped). When the coprocessor abandons the cache data of the memory address, the bit of the grid of the memory address includes the D bias state (i.e., the bit value of the bit is maintained).

[0055] When the bit of the grid needs to be flipped from the H bias state to the D bias state, the coprocessor needs to send a first refresh request to the main processor through the cache access device to request the coprocessor to abandon the cache data of the cache space corresponding to the bit. After the execution of the first refresh request is completed, the cache access of the coprocessor can enter the search of the second cache management table. In addition, cache requests that do not belong to the same bit belong to different cache request queues.

[0056] like Figure 5A As shown, the cache request queue includes request 0, request 1, request 2 and request 3. Request 0 and request 1 are cache requests initiated by the coprocessor GPU for the target memory address, request 2 is a cache request initiated by the main processor CPU for the target memory address, and request 3 is a cache request initiated by the main processor CPU or the coprocessor GPU for other memory addresses.

[0057] For example, if the bit where the target virtual address in the HDM table is located indicates that cache data exists in the cache space of the CPU, request 2 is determined to be dequeued from the cache request queue with priority, that is, request 2 can overtake request 0 and request 1. Without loss of generality, if the first cache state indicates that cache data exists in the first cache space, the cache request initiated by the main processor in the cache request queue is determined to be dequeued with priority.

[0058] For another example, request 3 is a cache request initiated by the main processor CPU or the coprocessor GPU for other memory addresses. There is no cache data consistency issue between the cache requests of other memory addresses and the cache requests of the target memory address. Request 3 can overtake request 0, request 1 and request 2, that is, request 3 is dequeued before request 0, request 1 and request 2; alternatively, request 3 may not overtake request 0, request 1 and request 2, that is, request 0, request 1 and request 2 are dequeued before request 3.

[0059] like Figure 5B As shown, the CPU first refresh request is an example of the first refresh request. If the bit where the target virtual address in the HDM table is located indicates that there is cache data in the cache space of the CPU, then before executing the cache request, request 0 or request 1 from the coprocessor GPU needs to refresh the cache data in the cache space of the main processor CPU to the target memory address in the memory space. Therefore, the CPU first refresh request is dequeued in the cache request queue, that is, the CPU first refresh request overtakes request 0 and request 1. Without loss of generality, if the first cache state indicates that there is cache data in the first cache space, a first refresh request is sent to the main processor, and the first cache state of the memory address range where the target memory address is located is changed, wherein the first refresh request is used to refresh the cache data in the memory address range where the target memory address is located in the first cache space to the memory address range. That is, the cache data consistency between the first cache space of the main processor and the second cache space of the coprocessor is improved by the execution of the first refresh request. It should be understood that the first refresh request is used to refresh the cache data in the first cache space of the memory address range where the target memory address is located to the memory address range, that is, the cache access device refreshes the data of all memory addresses in the memory address range where the memory address is located through the first refresh request, that is, the memory addresses of the existing cache data all refresh the cache data to the corresponding memory addresses, and then, in the first cache management table, the bit of the first cache state corresponding to the memory address range is changed to the memory address in the memory address range to indicate the non-existent cache data. In other words, there may be cache data of the memory address for which no cache request has been initiated that is refreshed to the corresponding memory address in the memory space, and when the main processor initiates a cache request for the memory address, it is necessary to directly access the corresponding memory address in the memory space, rather than the cache data corresponding to the memory address. Therefore, the smaller the memory address range represented by each bit in the first cache data table, the less data is refreshed by each first refresh request of the main processor, and the better the consistency of the cache data.

[0060] Furthermore, if Figure 5C As shown, request 2 and request 3 are initiated by the CPU and sent to the cache access device 200, and request 0 and request 1 are initiated by the GPU and sent to the cache access device 200. For request 3, the first cache state of the memory address of request 3 (e.g., the above-mentioned "other memory address") indicates that there is no cache data in the first cache space (e.g., the grid corresponding to the memory address of request 3 in the HDM table indicates the D bias state), and if the second cache state indicates that there is cache data in the corresponding second cache space, the cache access device 200 accesses the cache data in the second cache space of the GPU as a response to request 3, and then returns the response to request 3 to the CPU.

[0061] That is to say, if the first cache state indicates that there is no cache data in the first cache space, the second cache management unit determines whether the second cache state indicates that there is cache data in the corresponding second cache space; if so, the cache access execution unit executes the cache request initiated by the main processor for the second cache space of the coprocessor.

[0062] It should be understood that in this example, request 3 initiated by the CPU is queued after request 0 and request 1 initiated by the GPU and request 2 initiated by the CPU, and the memory address for request 3 may be different from the memory address for request 0, request 1, and request 2. In this case, request 3 may be dequeued with priority over request 0, request 1, and request 2. Without loss of generality, the execution order between cache requests for different memory addresses is not limited.

[0063] Furthermore, before executing request 0, request 1, and request 2, the first cache state of the memory address (e.g., the target memory address) indicates that there is cache data in the first cache space (e.g., the grid corresponding to the memory address in the HDM table indicates the H bias state). It should be understood that request 0 and request 1 are initiated by the CPU. In order to maintain the consistency of the cache data in the first cache space of the CPU and the cache data in the second cache space of the GPU, the cache data in the first cache space needs to be refreshed to the memory address. If request 0 or request 1 is dequeued first (i.e., the request is executed first) compared to request 2 initiated by the CPU, an interlocking phenomenon will occur between request 2 and request 0 or request 1. Therefore, in this example, the access to request 2 is executed first, and after the access to request 2 is completed, the first refresh request for the first cache space of the GPU is executed. It should be understood that the cache access device 200 may initiate the first refresh request before executing the first refresh request, that is, after the access to request 2 is completed, or before the access to request 2 is completed. If the first refresh request is initiated before the access to request 2 is completed, the efficiency of the first refresh request is improved while ensuring the reliability of each request dequeue. Specifically, the cache access device 200 executes the access of request 2 for the memory space (DDR or HBM) of the memory 300, and returns the access response to the CPU. Without loss of generality, if the first cache state indicates that there is cache data in the first cache space, the cache access execution unit executes the cache request initiated by the main processor for the data in the memory space of the memory 300 at the target memory address, thereby avoiding the interlocking between the cache request of the main processor and the cache request of the coprocessor, and improving the reliability of cache management.

[0064] Furthermore, the execution order of request 0 or request 1 initiated by the GPU is not limited. Figure 5CIn the example, the cache request queue is dequeued according to the enqueue order of request 0 and request 1. That is, generally, the enqueue order and dequeue order of cache requests sent by the same processor type (i.e., the main processor and the coprocessor are different processor types) for the same memory address can be the same, so as to improve the management efficiency of the cache request queue. Without loss of generality, if the second cache state indicates that there is cache data in the corresponding second cache space, the cache request initiated by the coprocessor in the cache request queue is determined to be dequeued first, thereby improving the reliability and efficiency of cache management.

[0065] Further, when executing request 0, the first refresh request for the first cache space of the GPU has been completed, and the first cache state of the memory address indicates that there is no cache data in the first cache space (for example, the grid corresponding to the memory address in the HDM table indicates the D bias state). In this case, for request 0, there is no cache data in the cache line of the second cache space corresponding to the memory address, and the cache access device 200 executes the access of request 0 for the memory space (DDR or HBM) of the memory 300, and returns the access response to the GPU, and caches the accessed data in the second cache space of the processing unit that initiates request 0 in the coprocessor.

[0066] Further, when executing request 1, the first refresh request for the first cache space of the GPU has been completed, and the first cache state of the memory address indicates that there is no cache data in the first cache space (for example, the grid corresponding to the memory address in the HDM table indicates the D bias state). In this case, for request 1, the cache line of the second cache space corresponding to the memory address already has cache data, and the cache access device 200 executes the access of request 1 for the second cache space and returns the access response to the GPU. It should be understood that the data accessed from the memory by request 0 is cached in the second cache space in the processing unit that initiates request 0. When the processing unit that initiates request 1 is the same as the processing unit that initiates the request, the processing unit where the cache data is located is the local processing unit of request 1, and the cache access device of the local processing unit directly accesses the cache data in the second cache space. Alternatively, when the processing unit that initiates request 1 is different from the processing unit that initiates the request, the processing unit where the cache data is located is the remote processing unit of request 1, and the cache access device of the local processing unit that receives request 1 indirectly accesses the cache data through the cache access device of the remote processing unit where the cache data is located, and the cache access device of the remote processing unit executes access of request 1 on the second cache space of the remote processing unit.

[0067] In other embodiments, for a chip including multiple processing units (e.g., bare die), the memory space corresponding to the local processing unit (e.g., DDR / HBM space) and the memory space corresponding to the remote processing unit are visible to the computing core of the local processing unit. The second cache management table includes multiple cache lines, and the cache attributes of each cache line include at least the memory address range corresponding to the cache data of the cache line, and the second cache state of whether the cache line has cache data in the corresponding second cache space. In other words, the second cache management table characterizes the correspondence between the memory addresses in the memory space shared by multiple processing units and the multiple processing units through the cache attributes of multiple cache lines.

[0068] In other words, the memory space is expanded, the chip design bottleneck of the coprocessor is solved, and the computing power of the coprocessor is significantly improved. When the coprocessor performs data calculations, the data required can be cached in the second cache space (for example, LLC) of the local processing unit, avoiding the local processing unit's computing core repeatedly accessing across processing units during the data reading and writing process, and the increase in data access delay caused by access to the memory space.

[0069] In addition, when the computing cores on different processing units share a memory space, data at the same memory address will be cached in the LLCs of different processing units. The second cache management table represents the correspondence between the memory addresses in the memory space shared by multiple processing units and the multiple processing units through the cache attributes of multiple cache lines, thereby solving the problem of cache data consistency between different processing units.

[0070] It should be understood that for the second cache management table, the cache access device in the processing unit and the corresponding second cache space belong to the processing unit, and the number of cache lines of the processing unit is determined for the internal cache access device, for example, the number of cache lines is 1024.

[0071] It should also be understood that each cache access device monitors the second cache state of the second cache space of the local processing unit at the same time, and also monitors the second cache state of the second cache space of the remote processing unit. In the case where the coprocessor is composed of two processing units, the second cache space of processing unit 0 is LLC0, and the second cache space of processing unit 1 is LLC1. Therefore, LLC0 and LLC1 cache DDR0 / HBM0 address spaces and are different from each other, and each cache access device can monitor 2*1024=2048 cache lines. In some examples, the second cache management table stored in the cache access device of each processing unit is the same, and if the second cache status of the corresponding cache line of the local processing unit that initiates the cache request indicates that cache data exists in the second cache space, the cache access device that receives the cache request executes the cache request access to the second cache space of the local processing unit; if the second cache status of the corresponding cache line of the local processing unit that initiates the cache request indicates that cache data does not exist in the second cache space, and the second cache status of the corresponding cache line of the remote processing unit indicates that cache data exists in the second cache space, the cache access device that receives the cache request in the local processing unit forwards the cache request to the cache access device of the remote processing unit, so as to execute the cache request access to the second cache space of the remote processing unit through the cache access device of the remote processing unit.

[0072] like Figure 5D As shown, the cache attributes of the cache line include memory address ranges such as virtual address / physical address, the hit status of each of N processing units (in this example, N=2), and the available status of the cache data of the cache line. That is, the second cache state includes the hit status and the available status of the cache line. It should be understood that multiple processing units correspond to multiple hit states, and when one of the multiple hit states is a valid hit, the other hit states are non-hit, that is, there is cache data in the second cache space of one processing unit of the multiple processing units, and the same cache data does not exist in the second cache space of other processing units, so as to ensure the consistency of cache data between different processing units.

[0073] In the cache access device 200, the second query unit 222 can determine whether LLC0 or LLC1 has a local data cache by comparing memory addresses such as virtual addresses / physical addresses.

[0074] After the cache request initiated by the main processor is processed by the first cache management unit, the cache line of the second cache management table is queried according to the cache request. If there is cache data in the local second cache space, the data backup is directly obtained from the local second cache space; if there is cache data in the remote second cache space, the data backup is directly obtained from the remote second cache space; if there is no cache data in multiple second cache spaces, the data is obtained from the memory address of the memory space.

[0075] After the cache request initiated by the local processing unit of the coprocessor is processed by the first cache management unit, the cache line of the second cache management table is queried according to the cache request. If there is cache data in the second cache space of the remote location, the cache data is directly obtained from the second cache space of the remote location; if there is no cache data in multiple second cache spaces, the data is obtained from the memory address of the memory space.

[0076] Furthermore, when the available state in the cache attribute is unavailable, other cache requests have been executed by the cache access device in the memory address range where the memory address is located, and subsequent cache access to the cache line of the same memory address must be suspended. That is, when the cache request is returned from the memory space, the available state is changed to available. The memory address range is an address management granularity that is not less than the memory address. Without loss of generality, if the hit state of the cache line indicates a hit, and the available state of the cache data of the cache line indicates available, that is, the cache data of all memory addresses in the memory address range exist in the second cache space of a processing unit, then the cache access execution unit executes the cache request access for the cache line. If the hit state of the cache line indicates a miss, that is, the cache data of all memory addresses in the memory address range where the memory address is located do not exist in the second cache space of any processing unit, then the cache access execution unit executes the cache request access for the target memory address. If the hit status of the cache line indicates a hit, and the available status of the cache data of the cache line indicates unavailable, that is, the cache data of all memory addresses in the memory address range where the memory address is located does not exist in the second cache space of any processing unit, and the request for the data of all memory addresses in the memory address range of the cache line has been initiated to the memory, then the cache access execution unit suspends the execution of the cache request access until the available status of the cache data of the cache line indicates available. It should be understood that the hit status of the processing unit is triggered when the cache request is issued from the cache access device, and multiple processing units greatly improve the scalability of the cache usage of the coprocessor.

[0077] It should be understood that the memory address range of the cache lines in the second cache management table is not greater than the memory address range of the first cache management table, that is, the address management granularity in the second cache management table is not greater than the address management granularity in the first cache management table.

[0078] When different processing units access cache data based on the second cache space, taking two processing units as an example, the cache request of the second cache space LLC0 is sent to the cache access device, and the cache access device queries that LLC1 of the second cache space has cache data cache, and the cache access device forwards the cache data access to the second cache space LLC1.

[0079] Then, the second cache space LLC1 executes the second refresh request, and subsequently receives a cache request with the same address from the computing core, and then receives a cache access with the same address forwarded by the cache access device. The second cache space LLC1 waits for the completion of the second refresh request. After receiving the completion return of the second refresh request, the second cache space LLC1 directly reads data from the memory space and returns it to the second cache space LLC0. Then, the cache access device sequentially executes the cache request of the second cache space LLC1.

[0080] That is, multiple processing units in the coprocessor share each cache line in the second cache management table. When the second cache state of the cache line in the second cache management table of a processing unit changes, the processing unit sends a synchronization instruction to the cache access devices of the remaining processing units through its own cache access device via the internal bus in the coprocessor to synchronize the change of the second cache state to the remaining processing units. That is, by sending the synchronization instruction, the consistency of the second cache management table of each processing unit is maintained.

[0081] Further, when a processing unit includes multiple processing sub-units (e.g., SM), the multiple processing sub-units can share the third cache management table of the processing unit, and the cache attributes of each cache line of the third cache management table include memory address ranges such as virtual address / physical address, third cache states of the third cache spaces of the M processing sub-units, and each third cache state includes a hit state of the cache line and an available state of cache data of the cache line. The address management granularity of the third cache management table is not greater than the address management granularity of the second cache management table.

[0082] It should be understood that multiple processing sub-units correspond to multiple hit states. When one of the multiple hit states is a valid hit, the other hit states are non-hit states. That is, there is cache data in the third cache space of one processing sub-unit of the multiple processing sub-units, and the same cache data does not exist in the third cache space of other processing sub-units, so as to ensure the consistency of cache data between different processing sub-units.

[0083] The following will be combined Figure 6 The cache access methods according to other embodiments of the present invention are described. Figure 6 The cache access method may be executed by a cache access device. Specifically, the cache access method includes:

[0084] S610: According to the target memory address in the cache request initiated by the coprocessor, query the first cache management table to obtain a first cache state corresponding to the target memory address, wherein the first cache management table indicates a first cache state of cached data in the first cache space of the main processor for each memory address range of the memory space.

[0085] S620: If the first cache status indicates that there is no cache data in the first cache space, the second cache management unit searches the second cache management table according to the cache request initiated by the coprocessor to obtain the second cache status of the cache line corresponding to the target memory address and the coprocessor, wherein the second cache management table indicates the second cache status of whether there is cache data in multiple second cache spaces in the coprocessor at different memory addresses.

[0086] S630: Execute access to the cache request according to the second cache state.

[0087] In the scheme of the embodiment of the present invention, the first cache management unit implements consistency management of the first cache space of the main processor and multiple second cache spaces between the coprocessor through the first cache management table, and decouples the first cache space from the second cache space. In addition, the second cache management unit implements consistency management of the multiple second cache spaces of the coprocessor through the second cache management table, and realizes decoupling between the multiple second cache spaces. Therefore, in a heterogeneous computing system including a main processor and a coprocessor, cache data consistency is improved while maintaining data interaction efficiency.

[0088] In other embodiments, the cache access method further includes: obtaining a cache request queue for a target memory address; and determining a cache request initiated by a processor whose cache status corresponding to the target memory address indicates the presence of the cache data to be dequeued from the cache request queue with priority.

[0089] In other embodiments, a cache request initiated by a processor whose cache status corresponding to the target memory address indicates that the cache data exists is determined to be prioritized for dequeuing from a cache request queue, including: if the first cache status indicates that cache data exists in the first cache space, then the cache request initiated by the main processor in the cache request queue is determined to be prioritized for dequeuing; if the second cache status indicates that cache data exists in the corresponding second cache space, then the cache request initiated by the coprocessor in the cache request queue is determined to be prioritized for dequeuing.

[0090] In some other embodiments, the first cache management table includes a plurality of bits, each bit indicates a different memory address range, and the bit value of each bit indicates a first cache state of the memory address range corresponding to the bit.

[0091] In other embodiments, the second cache management table includes multiple cache lines, and the cache attributes of each cache line include at least a memory address range corresponding to the cache data of the cache line, and whether the cache line has a second cache state of the cache data in the corresponding second cache space.

[0092] In some other embodiments, the second cache state includes a hit state of the cache line and an available state of the cache data of the cache line. The cache access method further includes: if the hit state of the cache line indicates a hit, and the available state of the cache data of the cache line indicates available, then executing the access of the cache request for the cache line; if the hit state of the cache line indicates a miss, then executing the access of the cache request for the target memory address; if the hit state of the cache line indicates a hit, and the available state of the cache data of the cache line indicates unavailable, then suspending the execution of the access of the cache request until the available state of the cache data of the cache line indicates available.

[0093] In other embodiments, the cache access method also includes: if the first cache state indicates that cache data exists in the first cache space, sending a first refresh request to the main processor and changing the first cache state of the memory address range where the target memory address is located, wherein the first refresh request is used to refresh the cache data in the first cache space to the memory address range where the target memory address is located.

[0094] In other embodiments, the cache access method also includes: querying the first cache management table according to the target memory address of the cache request initiated by the main processor to obtain a first cache state corresponding to the target memory address; if the first cache state indicates that there is cache data in the first cache space, executing the cache request initiated by the main processor for the data of the target memory address in the memory space.

[0095] In other embodiments, the cache access method also includes: if the first cache status indicates that there is no cache data in the first cache space, then determining whether the second cache status indicates that there is cache data in the corresponding second cache space; if so, executing the cache request initiated by the main processor for the second cache space of the coprocessor.

[0096] It should be understood that the specific implementation of each step in the cache access method can refer to the corresponding description of the corresponding unit in the embodiment of the cache access device above, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the steps of the cache access method described above can refer to the corresponding process description of each unit of the cache access device, which will not be repeated here.

[0097] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant regulations and standards, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0098] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present invention can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0099] The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., a random access memory (RAM), a read-only memory (ROM), a flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0100] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present invention.

[0101] The above implementation methods are only used to illustrate the embodiments of the present invention, and are not limitations of the embodiments of the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention. The patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A cache access device, comprising: A first cache management unit queries a first cache management table according to a target memory address in a cache request initiated by the coprocessor to obtain a first cache state corresponding to the target memory address, wherein the first cache management table indicates a first cache state of whether there is cached data in a first cache space in the main processor for each memory address range of the memory space; a second cache management unit, wherein if the first cache status indicates that cache data does not exist in the first cache space, the second cache management unit searches a second cache management table according to a cache request initiated by the coprocessor to obtain a second cache status of a cache line corresponding to the target memory address and the coprocessor, wherein the second cache management table indicates a second cache status of cache data in multiple second cache spaces in the coprocessor at different memory addresses; A cache access execution unit executes the access of the cache request according to the second cache state.

2. The cache access device according to claim 1, wherein: The cache access device also includes a queue management unit, which manages a cache request queue for a target memory address and determines a cache request initiated by a processor whose cache status corresponding to the target memory address indicates that the cache data exists as a priority to be dequeued from the cache request queue.

3. The cache access device according to claim 2, wherein: The queue management unit is specifically used for: If the first cache state indicates that cache data exists in the first cache space, determining the cache request initiated by the main processor in the cache request queue to be dequeued with priority; If the second cache status indicates that cache data exists in the corresponding second cache space, the cache request initiated by the coprocessor in the cache request queue is determined to be dequeued with priority.

4. The cache access device according to claim 1, wherein: The first cache management table includes a plurality of bits, each bit indicates a different memory address range, and the bit value of each bit indicates a first cache state of the memory address range corresponding to the bit.

5. The cache access device according to claim 1, wherein: The second cache management table includes multiple cache lines, and the cache attributes of each cache line include at least a memory address range corresponding to the cache data of the cache line and whether the cache line has a second cache state of the cache data in the corresponding second cache space.

6. The cache access device according to claim 5, wherein: The second cache state includes a hit state of the cache line and an available state of cache data of the cache line; The cache access execution unit is specifically used for: If the hit state of the cache line indicates a hit, and the available state of the cache data of the cache line indicates available, then performing access to the cache request for the cache line; If the hit status of the cache line indicates a miss, performing access to the cache request for the target memory address; If the hit status of the cache line indicates a hit, and the available status of the cache data of the cache line indicates unavailable, the access of the cache request is suspended until the available status of the cache data of the cache line indicates available.

7. The cache access device according to claim 1, wherein: The first cache management unit is further used for: If the first cache state indicates that cache data exists in the first cache space, a first refresh request is sent to the main processor, and the first cache state of the memory address range where the target memory address is located is changed, wherein the first refresh request is used to refresh the cache data in the first cache space in the memory address range where the target memory address is located to the memory address range.

8. The cache access device according to claim 1, wherein: The first cache management unit is further used for: According to the target memory address of the cache request initiated by the main processor, query the first cache management table to obtain a first cache state corresponding to the target memory address; If the first cache state indicates that cache data exists in the first cache space, the cache access execution unit executes a cache request initiated by the main processor for the data at the target memory address in the memory space.

9. The cache access device according to claim 1, wherein: The second cache management unit is further used for: If the first cache state indicates that cache data does not exist in the first cache space, determining whether the second cache state indicates that cache data exists in the corresponding second cache space; If yes, the cache access execution unit executes the cache request initiated by the main processor for the second cache space of the coprocessor.

10. A coprocessor comprising: Multiple processing units, each processing unit includes: The computing unit initiates a cache request; A cache access device according to any one of claims 1 to 9.

11. A heterogeneous computing system, comprising: Main processor; The coprocessor according to claim 10; The interconnection bus is connected between the main processor and the coprocessor.

12. A cache access method, comprising: According to the target memory address in the cache request initiated by the coprocessor, query the first cache management table to obtain a first cache state corresponding to the target memory address, wherein the first cache management table indicates a first cache state of whether there is cached data in the first cache space of the main processor for each memory address range of the memory space; If the first cache status indicates that cache data does not exist in the first cache space, the second cache management unit searches a second cache management table according to a cache request initiated by the coprocessor to obtain a second cache status of a cache line corresponding to the target memory address and the coprocessor, wherein the second cache management table indicates a second cache status of cache data in multiple second cache spaces in the coprocessor at different memory addresses; Accessing the cache request is performed according to the second cache state.

Images