Memory access request merging strategy for MCM-GPU

Abstract

The invention discloses a memory access request merging strategy for MCM-GPU. The memory access request merging strategy comprises the following steps: setting a coalescer comprising a PC filter and a merging table; in response to execution of the program, judging whether an SM initiates a memory access request for accessing a remote memory; if yes, judging whether the memory access request has high data locality or not; if not, sending the memory access request to an inter-module network; if yes, sending the memory access request to a merge table, and searching the merge table by taking a memory address in the memory access request as a search key; if the search is hit, adding the ID of the SM which initiates the memory access request to an entry of a merge table which records a corresponding memory address; and if the search is not hit, creating a new entry in the merge table, recording the related information of the memory access request into the new entry, and sending the memory access request to the inter-module network. The method can reduce the frequency and overhead of accessing the far-end memory, reduce the memory access delay, improve the utilization rate of the merge table, and reduce the hardware overhead.

Description

Technical Field

[0001] The present invention relates to the field of GPU technology, and in particular to a memory access request merging strategy for MCM-GPU. Background Art

[0002] With the continuous development of GPU (Graphics Processing Unit) technology, the number of GPU cores has been increasing, and the computing power has been increasing. For example, from Fermi to Volta architecture, the number of streaming multi-processors (SM) in GPU has increased from 14 to 80. Due to the limitations of manufacturing process and chip size, it is becoming increasingly difficult to integrate a large number of SMs on a single chip. In order to further improve the performance of GPU and avoid the constraints of chip size, a GPU architecture designed using a multi-chip module (MCM) method is currently proposed. The multi-chip module design is a discrete design in which the processor package contains multiple smaller cores, each of which contains only a small part of the resources of a traditional single-chip system-on-a-chip (SoC). It is easier and cheaper to manufacture a single core, and these cores are connected together through fast, high-bandwidth chip package internal interconnects to provide more computing resources. The MCM-GPU architecture is a high-performance computing architecture that improves computing performance by integrating multiple modules in one package. Compared with traditional high-throughput processor architectures, MCM-GPU has higher computing density, lower power consumption and better scalability. MCM-GPU improves computing performance by increasing the number of modules and can use more efficient interconnection technology without increasing the system footprint.

[0003] refer to Figure 1 , Figure 1A schematic diagram of the structure of an existing MCM-GPU is provided as an example. The MCM-GPU includes four GPU modules (GPU modules, GPMs). Each GPU module includes multiple SMs and a private L1 cache (L1 Cache) of the SM. The SM is connected to the GPU module memory system through an Xbar (cross switch). The GPU module memory system includes an L2 cache (L2 Cache) and a DRAM (Dynamic Random Access Memory) partition on the local memory side. All DRAM partitions provide a globally shared memory address space that can be shared and accessed by all GPU modules. In order to maximize the bandwidth and capacity of multiple DRAM partitions, memory addresses are fine-grained interleaved in all physical DRAM partitions. Xbar routes memory access to the correct location (local or remote L2 cache line) according to the physical address. Multiple Xbars also form a modular intra-package ring or mesh interconnection network. The L2 cache is a memory-side cache that only caches data from the local DRAM partition. Therefore, each cache line has only one location, and cache consistency does not need to be maintained between L2 cache lines. In the MCM-GPU architecture, a centralized CTA scheduler is usually used to globally schedule CTAs to be executed on SMs in a polling manner. The MCM-GPU memory system is a NUMA architecture. When accessing the memory on other GPU modules, there will be additional latency overhead, including the latency of moving data within the local GPU module to the edge of the chip, the serialization and deserialization latency on the interconnected GPU module link, and the wire transmission latency to the next GPU module.

[0004] In MCM-GPU, for a certain GPU module, its internal SM will generate a large amount of memory accesses sent to the remote GPU module to access the remote memory during the execution of the GPU program. Compared with accessing local memory, accessing remote memory will bring great memory access delays and memory access conflicts, resulting in the performance of MCM-GPU not being able to continue to improve with the increase in the number of GPU modules. In addition, due to the influence of memory access overhead, the performance of MCM-GPU will even decrease with the increase in the number of GPU modules. Summary of the invention

[0005] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides a memory access request merging strategy for MCM-GPU.

[0006] The technical solution of the present invention is as follows:

[0007] A memory access request merging strategy for an MCM-GPU is provided, wherein the MCM-GPU includes a plurality of interconnected GPU modules, and the strategy includes the following steps:

[0008] A coalescer is arranged in each GPU module, and the coalescer is respectively connected to the stream multiprocessor and the cross switch in the GPU module, and the coalescer includes a PC filter and a merge table connected to each other, the PC filter is used to determine whether a memory access request has high data locality, the merge table is used to record the memory access request, and the merge table is provided with a plurality of entries, and each entry includes an address field, a stream multiprocessor list, a valid bit and a PC field;

[0009] In response to the execution of the program, determining in real time whether a stream multiprocessor initiates a memory access request to access the remote memory;

[0010] If a stream multiprocessor in a GPU module initiates a memory access request to access remote memory, it is determined whether the memory access request has high data locality;

[0011] If the memory access request does not have high data locality, the memory access request is sent to the inter-module network to access the corresponding remote memory;

[0012] If the memory access request has high data locality, the memory access request is sent to the merge table, and the merge table is searched using the memory address in the memory access request as the search key;

[0013] If the search is successful, the ID of the stream multiprocessor that initiated the memory access request is added to the stream multiprocessor list in the entry that records the corresponding memory address;

[0014] If the search misses, a new entry is created in the merge table, the memory access request information is recorded in the new entry, and the memory access request is sent to the inter-module network to access the corresponding remote memory;

[0015] Receive the return result of the memory access request, query the merge table according to the reply address of the return result, if there is a matching entry in the merge table, broadcast the return result to all stream multiprocessors recorded in the matching entry, if there is no matching entry in the merge table, send the return result to the stream multiprocessor that initiated the memory access request.

[0016] In some optional implementations, the strategy further includes:

[0017] A finite state machine including four states is designed in the PC filter to determine whether the memory access request has high data locality;

[0018] The four states of the finite state machine include: strongly unshared, weakly unshared, weakly shared and strongly shared;

[0019] The initial state of the finite state machine is strongly shared.

[0020] In some optional implementations, the finite state machine determines whether the memory access request has high data locality in the following manner:

[0021] After the coalescer receives the data response returned by the memory access request, it queries the merge table, obtains the stream multiprocessor list and PC field data corresponding to the memory access request in the merge table, and determines whether the number of stream multiprocessors recorded in the stream multiprocessor list is greater than the set threshold. If so, it determines that the memory access request associated with the program counter recorded in the PC field data has high data locality, and performs local state transition processing on the state of the finite state machine corresponding to the current program counter. If not, it determines that the memory access request associated with the program counter recorded in the PC field data does not have high data locality, and performs non-local state transition processing on the state of the finite state machine corresponding to the current program counter.

[0022] When the state of the finite state machine corresponding to the program counter corresponding to the received memory access request is strongly unshared or weakly unshared, the PC filter determines that the currently received memory access request does not have high data locality. When the state of the finite state machine corresponding to the program counter corresponding to the received memory access request is weakly shared or strongly shared, the PC filter determines that the currently received memory access request has high data locality.

[0023] In some optional implementations, when the search misses, if the merge table is full, a memory access request is directly sent to the inter-module network to access the corresponding remote memory.

[0024] The main advantages of the technical solution of the present invention are as follows:

[0025] The memory access request merging strategy for MCM-GPU of the present invention determines whether the memory access request has high data locality by setting a PC filter, adds the memory access request with high data locality to the set merging table and merges according to the memory address, directly sends the memory access request without high data locality to the inter-module network, and broadcasts the returned result to multiple stream multiprocessors or sends it to the stream multiprocessor specified by the memory access request according to the address information recorded in the merging table after the memory access request returns the result, which can reduce the number of accesses to remote memory and the overhead, reduce memory access conflicts, reduce memory access delays, improve the performance of MCM-GPU, and at the same time can avoid the merge table recording addresses that are only accessed by a single stream multiprocessor as much as possible, significantly improve the utilization rate of the merge table, reduce the required merge table capacity, and reduce hardware overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1A schematic diagram of the structure of an existing MCM-GPU provided as an example;

[0028] Figure 2 A flowchart of a memory access request merging strategy for MCM-GPU provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the structure of a coalescer provided in an embodiment of the present invention;

[0030] Figure 4 A combined representation diagram provided for an embodiment of the present invention;

[0031] Figure 5 A schematic diagram of a state switching method of a finite state machine provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] The technical solution provided by the embodiments of the present invention is described in detail below with reference to the accompanying drawings.

[0034] refer to Figure 2-4 The embodiment of the present invention provides a memory access request merging strategy for an MCM-GPU, wherein the MCM-GPU includes a plurality of interconnected GPU modules, and the strategy includes the following steps:

[0035] Step 1, setting a coalescer (AdCoalescer) in each GPU module, the coalescer is respectively connected to the stream multiprocessor (SM) and the crossbar switch (Xbar) in the GPU module, the coalescer includes a PC filter (PC Filter) and a merge table (Merge Table) connected to each other, the PC filter is used to determine whether a memory access request has high data locality, and the merge table is used to record the memory access request;

[0036] In an embodiment of the present invention, the merge table is provided with a plurality of entries, each entry including an address field, a stream multiprocessor list, a valid bit and a PC field;

[0037] In an embodiment of the present invention, in an entry in the merge table, the address field (Address) is used to record the memory address of the cache line that needs to be accessed in the memory access request, the stream multiprocessor list (SM List) is used to record the ID of the stream multiprocessor that requests the cache line in the address field, the valid bit (Valid Bit) is used to record whether the entry is active, and the PC field (PC) is used to record the program counter related to the memory address of the cache line in the address field.

[0038] Step 2, in response to the execution of the program, determining in real time whether a stream multiprocessor initiates a memory access request to access the remote memory;

[0039] In the embodiment of the present invention, the remote memory represents a local or remote L2 cache.

[0040] Step 3: if a stream multiprocessor in a GPU module initiates a memory access request to access a remote memory, determine whether the memory access request has high data locality;

[0041] In the embodiment of the present invention, a PC filter in a coalescer provided in a GPU module is used to determine whether a memory access request has high data locality.

[0042] In the embodiment of the present invention, if the memory address to be accessed in the memory access request is likely to be accessed concurrently by multiple streaming multiprocessors, it means that the memory access request has high data locality.

[0043] Step 4: If the memory access request does not have high data locality, the memory access request is sent to the inter-module network to access the corresponding remote memory;

[0044] Step 5: If the memory access request has high data locality, the memory access request is sent to the merge table, and the merge table is searched using the memory address in the memory access request as a search key;

[0045] Step 6: If the search is successful, the ID of the stream multiprocessor that initiated the memory access request is added to the stream multiprocessor list in the entry recording the corresponding memory address;

[0046] In the embodiment of the present invention, a search hit indicates that a memory address identical to the memory address in the memory access request exists in the information recorded in the merge table.

[0047] Step 7: If the search does not hit, a new entry is created in the merge table, the memory access request related information is recorded in the new entry, and the memory access request is sent to the inter-module network to access the corresponding remote memory;

[0048] In the embodiment of the present invention, a search miss indicates that the information recorded in the merge table does not contain a memory address that is the same as the memory address in the memory access request.

[0049] Step 8, receive the return result of the memory access request, query the merge table according to the reply address of the return result, if there is a matching entry in the merge table, broadcast the return result to all stream multiprocessors recorded in the matching entry, if there is no matching entry in the merge table, send the return result to the stream multiprocessor that initiated the memory access request.

[0050] In an embodiment of the present invention, when the return result of a memory access request is replied to the GPU module, the aggregator in the GPU module queries the merge table according to the reply address of the return result. If there is a matching entry in the merge table, the return result is broadcast to all stream multiprocessors recorded in the matching entry. If there is no matching entry in the merge table, the return result is sent to the stream multiprocessor that initiated the memory access request.

[0051] In an embodiment of the present invention, if there is a memory address in the information recorded in the merged table that is the same as the reply address of the returned result, it means that there is a matching entry in the merged table, and the entry with the same memory address is an entry that matches the returned result, otherwise it means that there is no matching entry in the merged table.

[0052] The memory access request merging strategy for MCM-GPU provided by the embodiment of the present invention determines whether the memory access request has high data locality by setting a PC filter, adds the memory access request with high data locality to the set merging table and merges it according to the memory address, directly sends the memory access request without high data locality to the inter-module network, and broadcasts the returned result to multiple stream multiprocessors or sends it to the stream multiprocessor specified by the memory access request according to the address information recorded in the merging table after the memory access request returns the result. The number and overhead of accessing remote memory can be reduced, memory access conflicts can be reduced, memory access latency can be reduced, and the performance of MCM-GPU can be improved. At the same time, the merge table can be avoided as much as possible to record the address that is only accessed by a single stream multiprocessor, which significantly improves the utilization rate of the merge table, reduces the required merge table capacity, and reduces hardware overhead.

[0053] Furthermore, in an embodiment of the present invention, the PC filter identifies memory access requests with high data locality by predicting memory addresses that may be accessed concurrently by multiple streaming multiprocessors.

[0054] refer to Figure 5Since the data access mode of MCM-GPU is affected by multiple factors such as application algorithm, CTA scheduling between streaming multiprocessors and fine-grained scheduling within streaming multiprocessors, and considering these characteristics of high-throughput processor applications, in an embodiment of the present invention, a finite state machine including four states is designed in the PC filter to determine whether a memory access request has high data locality, wherein the four states of the finite state machine include: strongly not shared, weakly not shared, weakly shared and strongly shared.

[0055] Further, in an embodiment of the present invention, the finite state machine determines whether the memory access request has high data locality in the following manner:

[0056] After the coalescer receives the data response returned by the memory access request, it queries the merge table, obtains the stream multiprocessor list and PC field data corresponding to the memory access request in the merge table, and determines whether the number of stream multiprocessors recorded in the stream multiprocessor list is greater than the set threshold. If so, it determines that the memory access request associated with the program counter recorded in the PC field data has high data locality, and performs local state transition processing on the state of the finite state machine corresponding to the current program counter. If not, it determines that the memory access request associated with the program counter recorded in the PC field data does not have high data locality, and performs non-local state transition processing on the state of the finite state machine corresponding to the current program counter.

[0057] When the state of the finite state machine corresponding to the program counter corresponding to the received memory access request is strongly unshared or weakly unshared, the PC filter determines that the currently received memory access request does not have high data locality. When the state of the finite state machine corresponding to the program counter corresponding to the received memory access request is weakly shared or strongly shared, the PC filter determines that the currently received memory access request has high data locality.

[0058] In an embodiment of the present invention, local state transition processing means converting the state of a finite state machine from a current state to a next-level state, for example, from strongly non-sharing to weakly non-sharing, from weakly non-sharing to weakly sharing, from weakly sharing to strongly sharing, or keeping strongly sharing unchanged; non-local state transition processing means converting the state of a finite state machine from a current state to a previous state, for example, from weakly non-sharing to strongly non-sharing, from weakly sharing to weakly non-sharing, from strongly sharing to weakly sharing, or keeping strongly non-sharing unchanged.

[0059] Furthermore, in the embodiment of the present invention, the initial state of the finite state machine is set to strong sharing.

[0060] Furthermore, in an embodiment of the present invention, when targeting an application scenario of a single GPU application, if the state of the finite state machine corresponding to the program counter is strongly unshared, the state of the finite state machine corresponding to the program counter will no longer undergo state transition and will always remain strongly unshared.

[0061] It should be noted that in the attached Figure 5 In, T NL and T L They represent non-local state transition and local state transition respectively.

[0062] Furthermore, in the embodiment of the present invention, in step 7, if the merge table is full, the memory access request is directly sent to the inter-module network to access the corresponding remote memory.

[0063] In an embodiment of the present invention, since the capacity of the merge table is limited, if a memory access request has high data locality, when the merge table is searched using the memory address in the memory access request as the search key, if the search does not hit, and the merge table is full and a new entry cannot be created, the memory access request is not recorded and the memory access request is directly sent to the inter-module network to access the corresponding remote memory.

[0064] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, "front", "back", "left", "right", "upper" and "lower" in this article are all referenced to the placement state shown in the accompanying drawings.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A memory access request merging strategy for MCM-GPU, characterized in that: The MCM-GPU includes a plurality of interconnected GPU modules, and the strategy includes the following steps: A coalescer is arranged in each GPU module, and the coalescer is respectively connected to the stream multiprocessor and the cross switch in the GPU module, and the coalescer includes a PC filter and a merge table connected to each other, the PC filter is used to determine whether a memory access request has high data locality, the merge table is used to record the memory access request, and the merge table is provided with a plurality of entries, and each entry includes an address field, a stream multiprocessor list, a valid bit and a PC field; In response to the execution of the program, determining in real time whether a stream multiprocessor initiates a memory access request to access the remote memory; If a stream multiprocessor in a GPU module initiates a memory access request to access remote memory, it is determined whether the memory access request has high data locality; If the memory access request does not have high data locality, the memory access request is sent to the inter-module network to access the corresponding remote memory; If the memory access request has high data locality, the memory access request is sent to the merge table, and the merge table is searched using the memory address in the memory access request as the search key; If the search is successful, the ID of the stream multiprocessor that initiated the memory access request is added to the stream multiprocessor list in the entry that records the corresponding memory address; If the search misses, a new entry is created in the merge table, the memory access request information is recorded in the new entry, and the memory access request is sent to the inter-module network to access the corresponding remote memory; Receive the return result of the memory access request, query the merge table according to the reply address of the return result, if there is a matching entry in the merge table, broadcast the return result to all stream multiprocessors recorded in the matching entry, if there is no matching entry in the merge table, send the return result to the stream multiprocessor that initiated the memory access request.

2. The memory access request merging strategy for MCM-GPU according to claim 1, characterized in that: The strategy also includes: A finite state machine including four states is designed in the PC filter to determine whether the memory access request has high data locality; The four states of the finite state machine include: strongly unshared, weakly unshared, weakly shared and strongly shared; The initial state of the finite state machine is strongly shared.

3. The memory access request merging strategy for MCM-GPU according to claim 2, characterized in that: The finite state machine determines whether a memory access request has high data locality by: After the coalescer receives the data response returned by the memory access request, it queries the merge table, obtains the stream multiprocessor list and PC field data corresponding to the memory access request in the merge table, and determines whether the number of stream multiprocessors recorded in the stream multiprocessor list is greater than the set threshold. If so, it determines that the memory access request associated with the program counter recorded in the PC field data has high data locality, and performs local state transition processing on the state of the finite state machine corresponding to the current program counter. If not, it determines that the memory access request associated with the program counter recorded in the PC field data does not have high data locality, and performs non-local state transition processing on the state of the finite state machine corresponding to the current program counter. When the state of the finite state machine corresponding to the program counter corresponding to the received memory access request is strongly unshared or weakly unshared, the PC filter determines that the currently received memory access request does not have high data locality. When the state of the finite state machine corresponding to the program counter corresponding to the received memory access request is weakly shared or strongly shared, the PC filter determines that the currently received memory access request has high data locality.

4. The memory access request merging strategy for MCM-GPU according to claim 1, characterized in that: When the search misses, if the merge table is full, the memory access request is directly sent to the inter-module network to access the corresponding remote memory.

Images