Memory access request processing method and apparatus for general purpose graphics processor

Abstract

The application relates to a memory access request processing method and device for a general-purpose graphics processor. The method comprises the following steps: determining whether a memory access request of a first thread bundle for a target cache line hits in an L1 cache; if not, inquiring whether a local uncompleted miss record triggered by the first thread bundle exists in a thread bundle miss state holding register array; if not, inquiring whether a global uncompleted miss record triggered by other thread bundles for the target cache line exists in a global miss state holding register table; if yes, adding the first thread bundle into a waiting thread bundle set corresponding to the global record, waiting for data returned by an initiated downstream access; and after the data is returned, distributing the data to all thread bundles in the waiting thread bundle set through a broadcast mechanism.

Description

Technical Field

[0001] This application relates generally to the field of computers, and more specifically to a method and apparatus for processing memory access requests for a general-purpose graphics processor. Background Technology

[0002] A general-purpose graphics processing unit (GPGPU) is a processor architecture with highly parallel computing capabilities, widely used in scientific computing, artificial intelligence training, big data analysis, and other fields. In the GPGPU architecture, multiple warps execute in parallel, with each warp containing several threads working together to complete the computational task.

[0003] Modern GPGPUs employ a multi-level storage architecture to balance access speed and storage capacity. When the data required by a load instruction executed by a thread bundle is not in the L1 cache, a cache miss occurs, and the data needs to be retrieved from the next lower storage level.

[0004] To handle cache misses and hide memory access latency, GPGPU employs a MissStatus Holding Register (MSHR). MSHR records the status information of incomplete cache miss requests, including the address of the missing cache line, the thread bundle identifier from which the request originated, and the mask of the thread waiting for the data. When an L1 cache miss occurs, MSHR allocates an entry to record the miss request and initiates a memory access request to the next lower memory level. After the data is returned, MSHR writes the data back to the register file of the requesting thread bundle based on the recorded information and wakes up the waiting thread bundle, allowing it to resume execution. Summary of the Invention

[0005] This application provides a method and apparatus for processing memory access requests for general-purpose graphics processors. By introducing a two-level missing state recording mechanism, it realizes the merging of memory access requests across thread bundles, thereby reducing MSHR resource consumption and downstream storage layer bandwidth pressure.

[0006] According to one aspect of this application, a memory access request processing method for a general-purpose graphics processor is provided, characterized by comprising: determining whether a memory access request of a first thread bundle for a target cache line hits in the L1 cache; in response to determining that the memory access request of the first thread bundle for the target cache line misses in the L1 cache, querying the thread bundle miss status holding register array to see if there is a local incomplete missing record for the target cache line triggered by the first thread bundle; in response to determining that there is no local incomplete missing record in the thread bundle miss status holding register array, querying the global miss status holding register table to see if there is a global incomplete missing record for the target cache line triggered by other thread bundles; in response to determining that there is a global incomplete missing record for the target cache line triggered by other thread bundles in the global miss status holding register table, adding the first thread bundle to the waiting thread bundle set corresponding to the global incomplete missing record for the target cache line, and waiting for the return data of the access to the target cache line initiated to the subsequent memory level; after the data of the target cache line is returned, distributing the data simultaneously to all waiting thread bundles in the waiting thread bundle set for use through a broadcast mechanism.

[0007] According to another aspect of this application, a memory access request processing apparatus for a general-purpose graphics processor (GPU) is provided, comprising: a thread bundle missing status holding register array coupled to a load memory unit of the GPU and used to store local incomplete missing records of thread bundles for cache lines, the local incomplete missing records including an identifier of the thread bundle and the cache line requested by the thread bundle; a global missing status holding register table coupled to the thread bundle missing status holding register array and used to store global incomplete missing records for cache lines, the global incomplete missing records including address information of the cache line and a set of waiting thread bundles for the cache line; and a broadcast engine coupled to the global missing status holding register table and a register file of the GPU, and used to distribute return data of a target cache line to all waiting thread bundles in the global incomplete missing records of the target cache line via a broadcast mechanism, wherein the GPU is configured to perform the above method. Attached Figure Description

[0008] Figure 1 The illustration shows a flowchart of a memory access request processing method for a general-purpose graphics processor according to an embodiment of this application.

[0009] Figure 2 This is a structural block diagram of a memory access request processing apparatus for a general-purpose graphics processor according to an embodiment of this application.

[0010] Figure 3The illustration shows a schematic diagram of an example computing device in which the above-described methods and / or apparatus can be implemented according to an embodiment of this application. Detailed Implementation

[0011] The features and exemplary embodiments of various aspects of this application will now be described in detail. Numerous specific details are provided in the following detailed description to provide a thorough understanding of this application. However, it will be apparent to those skilled in the art that this application can be implemented without some of these specific details. The following description of embodiments is merely intended to provide a better understanding of this application by illustrating examples. This application is by no means limited to any specific configurations and algorithms described below, but rather covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of this application. Well-known structures and techniques are not shown in the accompanying drawings and the following description in order to avoid unnecessarily obscuring this application.

[0012] In a GPGPU parallel execution environment, multiple thread bundles may access the same cache line simultaneously or sequentially. When multiple thread bundles access the same cache line and all experience L1 cache misses, MSHR needs to merge these multiple miss requests. Traditional MSHR merging mechanisms typically operate at the request queue level. For multiple miss requests arriving simultaneously for the same cache line, they can be merged into a single MSHR entry and a single access can be initiated to the next lower-level storage.

[0013] However, traditional MSHR merging mechanisms have inherent time window limitations. Once the first thread's missing request has completed MSHR entry allocation and sent a memory access request to the downstream storage level, if a second thread arrives later and accesses the same cache line, the traditional mechanism typically cannot merge the second thread's request into the already initiated missing record. This is because the MSHR entry merging window is usually limited to the period before the request leaves the current processing queue; once the request has been sent downstream, the entry exits the mergeable state. In this case, the missing entry of the second thread will trigger new MSHR entry allocation and new downstream memory access requests, resulting in multiple redundant accesses to the same cache line.

[0014] Furthermore, in the traditional MSHR mechanism, MSHR allocation and occupancy occur from a thread-bundle-local perspective. When multiple thread bundles access the same cache line, even if the cache line already has incomplete missing records in the MSHR, subsequent thread bundles may still allocate independent MSHR entries. This thread-bundle-local approach to MSHR management leads to the possibility of the same cache line being recorded repeatedly by multiple MSHR entries, resulting in the ineffective use of scarce MSHR resources.

[0015] Traditional MSHR (Multiple Threads Hierarchical Retention) mechanisms lead to the following technical problems in scenarios where multiple threads concurrently access the same cache line: MSHR entries are duplicated, resulting in a waste of scarce MSHR resources; multiple downstream memory access requests are initiated for the same cache line, increasing the access pressure on on-chip networks, L2 caches, and dynamic random access memory; redundant memory access requests consume valuable storage bandwidth, impacting overall system performance and energy efficiency. The impact of these problems is particularly significant in application scenarios with high memory access conflicts, such as large-model tensor parallel training.

[0016] In view of the above problems, this application provides a multi-level MSHR address fusion scheme for GPGPU. This scheme uses the cache line address as the center, aggregating cache miss requests from multiple thread bundles for the same address into a single global miss record, and initiating only one downstream memory access for that address. In traditional MSHR mechanisms, MSHR is organized around "requests," and each thread bundle's miss request may be assigned an independent MSHR entry. Even for the same address, subsequent arriving requests are difficult to merge, resulting in the same address being repeatedly recorded by multiple MSHR entries, leading to multiple downstream accesses. This application, however, utilizes a global MSHR structure, indexed by the cache line address, to record the incomplete miss status for each address and the identifiers of all thread bundles waiting for that address. This ensures that regardless of the number of thread bundles or when they arrive, all miss requests for the same address are "fused" into the same global record, initiating only one access to the next storage level.

[0017] Specifically, the scheme includes a primary thread bundle MSHR array and a secondary global MSHR structure. The primary thread bundle MSHR array, indexed by thread bundle identifiers, records local missing information for each thread bundle. When a cache miss occurs, a local query is performed first. If no corresponding record is found locally, the secondary global MSHR structure is queried. If an incomplete missing record for the same cache line, triggered by another thread bundle, already exists in the secondary global MSHR structure, the identifier of the current thread bundle is added to the waiting thread bundle record for that record. No new MSHR entry needs to be allocated, nor is a new downstream access initiated. After the data is returned, it is simultaneously distributed to all thread bundles in the waiting thread bundle record via a broadcast mechanism for their use. This suppresses duplicate MSHR usage and reduces bandwidth pressure on lower storage layers.

[0018] The present application will now be described in further detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the application to these specific forms.

[0019] Figure 1 The illustration shows a flowchart of a memory access request processing method 100 for a general-purpose graphics processor according to an embodiment of this application. Figure 1 As shown, method 100 may include the following steps S101-S106. It will be understood that in other embodiments, method 100 may include more or fewer steps.

[0020] In step S101, it is determined whether the memory access request of the first thread bundle for the target cache line is hit in the L1 cache.

[0021] Specifically, when the Load Memory Unit (LSU) in the GPGPU receives a memory access request (e.g., a load instruction (LD), a store instruction (ST), a tensor load instruction issued by the tensor core, etc.) from a thread bundle (e.g., the first thread bundle), it first checks whether the target cache line data required by the request is already cached in the L1 cache. If a cache hit occurs, the data is read directly from the L1 cache, and the process ends. If a cache miss occurs, the process proceeds to step S102.

[0022] In step S102, in response to determining that the memory access request of the first thread bundle for the target cache line is missed in the L1 cache, the thread bundle missing state holding register array (thread bundle MSHR array) is queried to see if there is a local incomplete missing record triggered by the first thread bundle in the target cache line.

[0023] In embodiments of this application, the thread bundle MSHR array can be a pool of MSHR entries shared by all thread bundles. Each entry is marked as belonging to a specific thread bundle during allocation, thus logically constituting a thread bundle-private incomplete request record. Specifically, the array can include multiple MSHR entries. When a thread bundle (e.g., the first thread bundle) experiences a cache miss for the first time and needs to be recorded, a free entry is allocated to it from the shared pool. This entry can record relevant information about the miss, such as the target cache line address, thread bundle identifier (Warp ID), etc., thereby binding the entry to the first thread bundle. In this way, the entry becomes the local incomplete miss record of the first thread bundle.

[0024] In embodiments of this application, the thread bundle MSHR array can support queries indexed by thread bundle identifiers to quickly identify which MSHR entries a particular thread bundle currently occupies. For example, when the first thread bundle subsequently accesses the same cache line again, it can quickly locate the entry previously assigned to it by querying the array, thus knowing that it has an incomplete request for that cache line.

[0025] In this step, if a local record is found in the thread bundle MSHR array, it indicates that the thread bundle has already initiated a missing request for this cache line and has not yet returned data. In this case, we can wait for the request to return data without performing a global query. If no local record is found, proceed to step S103.

[0026] In step S103, in response to determining that there are no locally incomplete missing records in the thread bundle MSHR array, the Global Missing State Holding Register Table (GMEMT) is queried to see if there are any globally incomplete missing records for the target cache line triggered by other thread bundles.

[0027] In embodiments of this application, the GMEMT can be a content-addressable memory (CAM) based hardware structure or a hardware structure with parallel query capabilities. In embodiments of this application, the GMEMT can be a cross-thread bundle shared structure used to record all ongoing cache line miss requests triggered by any thread bundle. In one embodiment, the GMEMT can be physically located between the Load Memory Unit (LSU) within a streaming multiprocessor (SM) and the lower-level storage hierarchy (such as L2 cache), serving as a dedicated hardware unit for processing L1 cache miss requests. In another embodiment, the GMEMT can also be integrated into the L2 cache controller, with the L2 cache uniformly managing global miss records from multiple SMs. Those skilled in the art should understand that the specific physical location of the GMEMT is not a core limitation of this application. As long as the hardware unit can achieve the functions of "recording global incomplete miss requests using cache line addresses as keys, managing wait sets across thread bundles, and triggering broadcast distribution after data return," whether it is an independent module within an SM, a component of the L2 controller, or distributed across other storage levels, it falls within the protection scope of this application.

[0028] In embodiments of this application, the GMEMT may include multiple entries, each entry including at least: an address key value for uniquely identifying a cache line, and a set of waiting thread bundles for recording all thread bundles waiting for the cache line data to return. In one embodiment, the address key value may be the address information of the target cache line, for example, it may consist of the physical address and memory identifier of the target cache line. In one embodiment, the set of waiting thread bundles may be implemented as a thread bundle bitmap, where each bit in the bitmap may correspond to the identifier of a thread bundle.

[0029] In the embodiments of this application, each entry in GMEMT may also include other metadata, such as request type (e.g., load, store, atomic operation), data size, and other information.

[0030] In embodiments of this application, each entry in the GMEMT may also include a valid bit, such as a 1-bit flag, to indicate whether the entry is currently occupied by an ongoing (incomplete) global missing request. When the valid bit is a first logical value (e.g., "1"), it indicates that the entry is valid, and the address key value and waiting thread bundle set information it contains are valid; when the valid bit is a second logical value (e.g., "0"), it indicates that the entry is free and can be allocated to a new missing request. By setting the valid bit, the GMEMT can efficiently manage its limited entry resources. When data is returned from a subsequent storage level and distributed to all waiting thread bundles by the broadcast engine, the valid bit of the corresponding GMEMT entry is cleared, and the entry is released and can be used to record missing requests for subsequent cache lines.

[0031] In an embodiment of this application, at step S103, if no global record for the target cache line is found in GMEMT, a new global incomplete missing record can be created for the cache line at step S106. The new record may include the address information of the target cache line (e.g., LineAddr+BankID) and the initial set of waiting thread bundles (at least the first thread bundle). Then, an access request for the target cache line is initiated to the subsequent storage level. If a global record is found, the process proceeds to step S104.

[0032] In step S104, in response to determining that there is a globally incomplete missing record for the target cache line triggered by other thread bundles in the GMEMT, the first thread bundle is added to the waiting thread bundle set corresponding to the globally incomplete missing record for the target cache line.

[0033] In the embodiments of this application, if a global record of the target cache line already exists in GMEMT, it means that another thread bundle (which can be called the first-visit thread bundle) has already triggered a downstream access to the cache line, and the current first thread bundle does not need to initiate a new downstream request. It is only necessary to add the identifier of the first thread bundle to the waiting thread bundle set of the global record.

[0034] In embodiments of this application, the waiting thread bundle set can be implemented in the form of a thread bundle bitmap. Each bit in the bitmap corresponds to an identifier of a thread bundle, and the size of the bitmap can be determined by the number of concurrent thread bundles supported within the streaming multiprocessor (SM). Adding a new thread bundle to the set involves setting the bit position corresponding to that thread bundle to a first logical value (e.g., "1").

[0035] However, those skilled in the art should understand that the implementation of the waiting thread bundle set is not limited to a bitmap. In other embodiments of this application, the set can also be implemented using other data structures, as long as they can record a set of identifiers for waiting thread bundles. For example, in an alternative embodiment, the waiting thread bundle set can be a linked list of thread bundle identifiers. A GMEMT entry stores a head pointer of the linked list, pointing to the record of the first waiting thread bundle, and subsequent records are linked by pointers. When data returns, the broadcast engine traverses the linked list and wakes up each thread bundle in turn. In another alternative embodiment, the waiting thread bundle set can be a queue or array of fixed depth. A certain number of slots are reserved in the GMEMT entry for storing the identifiers of waiting thread bundles in sequence. When data returns, the broadcast engine reads all identifiers in the queue in sequence to wake up the thread bundles.

[0036] After the identifier of the first thread bundle is added to the set of waiting thread bundles in the global record, the first thread bundle can enter a waiting state and wait for the downstream access that has been initiated to return data.

[0037] In step S105, after the data of the target cache line is returned, the data is distributed to all waiting thread bundles in the waiting thread bundle set for use via a broadcast mechanism.

[0038] In embodiments of this application, when data from the target cache line returns from subsequent storage tiers (e.g., L2 cache, on-chip network (NoC), DRAM), GMEMT receives a data return signal. At this time, GMEMT's broadcast engine can simultaneously distribute the data to all waiting thread bundles based on, for example, a waiting thread bundle bitmap in the global record. Specifically, the distribution method could be to simultaneously write the data back to the register file corresponding to each waiting thread bundle and wake up these thread bundles to continue execution. After distribution is complete, the global record is cleared or marked as invalid.

[0039] In embodiments of this application, the broadcast engine may be a dedicated hardware unit coupled to the GMEMT, used to implement parallel distribution and thread wake-up after data return. In one embodiment, the broadcast engine may include parallel wake-up logic and distribution control logic.

[0040] Specifically, when data from the target cache line is returned from a subsequent storage level, GMEMT passes the corresponding global record (including the set of waiting thread bundles, such as a thread bundle bitmap) along with the returned data to the broadcast engine. The broadcast engine can then generate corresponding register file write enable signals in parallel within a single clock cycle based on the state of each bit in the set of waiting thread bundles, simultaneously writing the returned data to the register files of all waiting thread bundles. After the data write-back is complete, the broadcast engine sends a wake-up signal to the scheduler, indicating that these thread bundles are ready to continue execution, and are thus rescheduled for execution.

[0041] In embodiments employing thread-bound bitmaps, the broadcast engine's parallel wake-up logic can directly use the bitmap as a write enable mask to drive the write port of the register file, achieving true "simultaneous" distribution. After distribution is complete, the broadcast engine sends a completion signal to GMEMT, which clears the valid bits of the corresponding entry, releasing the global record for subsequent use.

[0042] The broadcast engine can be implemented in various ways, including but not limited to: a parallel decoder implemented with combinational logic circuits, a sequential dispatcher controlled by microcode, or a hybrid structure of both. Regardless of the specific implementation used, as long as it can achieve the function of "simultaneously providing return data to multiple thread bundles according to the set of waiting thread bundles", it falls within the protection scope of this application.

[0043] Those skilled in the art will understand that data distribution methods are not limited to register file write-back. In other embodiments of this application, the broadcast engine may also employ other distribution mechanisms, such as simultaneously writing data to the L1 cache to complete cache line filling; for streaming access, forwarding directly to the execution pipeline via a dedicated bypass path without writing to the cache; for tensor loading instructions, writing directly to shared memory; for texture access, sending to the texture filtering unit for processing before distribution; or simultaneously distributing data to multiple destinations (such as the L1 cache and register file). Any distribution path capable of simultaneously providing returned data to multiple waiting thread bundles falls within the protection scope of this application.

[0044] Figure 2 This is a structural block diagram of a memory access request processing apparatus 200 for a general-purpose graphics processor according to an embodiment of this application. The apparatus 200 can be integrated within the general-purpose graphics processor, for example, as a dedicated hardware unit between the load memory unit and the lower memory level. In another embodiment, the apparatus can also be implemented as a standalone hardware module, integrated with existing GPGPU architectures via a standard interface, for example, as an extension module of the L2 cache controller or an off-chip acceleration unit. Those skilled in the art should understand that regardless of the physical form in which the apparatus is implemented, as long as it includes the aforementioned functional units and implements cross-thread memory access request merging, it falls within the protection scope of this application.

[0045] like Figure 2 As shown, device 200 may include: a thread bundle MSHR array 210, a GMEMT 220, and a broadcast engine 230. Those skilled in the art will understand that the device may also include other auxiliary circuitry, such as control logic, a clock management unit, and test interfaces. For the sake of brevity, Figure 2 Not shown in the diagram. This device can be coupled and cooperate with the following units in a general-purpose graphics processor: Load Memory Unit (LSU), L1 cache, register file, and lower-level memory interface.

[0046] In embodiments of this application, the load storage unit receives memory access requests from different thread bundles (e.g., thread bundle A, thread bundle B). The unit first accesses the L1 cache to determine if the requested data is hit. If hit, the data is directly returned from the L1 cache to the register file of the requesting thread bundle; if miss, the request information (including the target cache line address, thread bundle identifier, etc.) is sent to the thread bundle MSHR array 210.

[0047] In embodiments of this application, the thread bundle MSHR array 210 can be a pool of MSHR entries shared by all thread bundles. Each entry is marked as belonging to a specific thread bundle during allocation, thus logically constituting a thread bundle-private local incomplete missing record. In embodiments of this application, the thread bundle MSHR array 210 can be a dual-port static random access memory physically distributed within the load storage unit. This memory is directly addressed using the thread bundle identifier as an index. Each thread bundle occupies an independent register array entry, and each entry can include at least a valid bit, a missing address field, and a global pointer field. This array can be used to quickly query whether a thread bundle already has an incomplete request for a specific cache line, using the thread bundle identifier as an index. When a request is received from, for example, an LSU, the thread bundle MSHR array 210 first queries whether a local record for the target cache line triggered by the requesting thread bundle exists. If it exists, the requesting thread bundle is notified to wait for the data corresponding to the local record to be returned; if it does not exist, the request is forwarded to the GMEMT 220 for a global query.

[0048] In embodiments of this application, GMEMT 220 can be a global structure shared across thread bundles, used to record all ongoing cache line miss requests triggered by any thread bundle. In a specific hardware implementation, GMEMT 220 can be a content-addressable memory (CAM), for example, located inside a streaming multiprocessor, between the load memory unit and the lower-level memory layer interface, supporting parallel queries using the cache line address (preferably combined with the bank identifier BankID) as the key. Each GMEMT entry may include: an address key, a waiting thread bundle set (e.g., a thread bundle bitmap), etc. When a query request is received from the thread bundle MSHR array 210, GMEMT 220 checks whether a valid global record exists for the target cache line: if it exists, the identifier of the current thread bundle is added to the waiting thread bundle set of the record, and the LSU is notified that no new downstream access needs to be initiated; if it does not exist, a new global record is created for the cache line, the waiting thread bundle set (containing the current thread bundle) is initialized, and a memory access request is initiated to the subsequent memory layer (such as L2 cache, NoC, DRAM) through the lower-level memory layer interface. The lower-level memory interface is responsible for communicating with internal or external memory tiers (L2 cache, on-chip network, DRAM, etc.) of the GPGPU chip, sending memory access requests and receiving returned data. When data returns from the lower-level memory tier, this interface passes the data and the corresponding GMEMT entry identifier to the broadcast engine 230.

[0049] In the embodiments of this application, the broadcast engine 230 can be coupled to a dedicated hardware unit of the GMEMT 220 and the register file. Its core function is to implement parallel distribution and thread wake-up after data return. The broadcast engine internally includes parallel wake-up logic and distribution control logic. When it receives the returned data and its corresponding set of waiting thread bundles (e.g., a thread bundle bitmap), the broadcast engine can generate register file write enable signals in parallel within a single clock cycle, writing the data simultaneously to the register files of all waiting thread bundles. After the data write-back is complete, the broadcast engine sends a wake-up signal to the scheduler (not shown), indicating that these thread bundles are ready to continue execution. At the same time, the broadcast engine sends a completion signal to the GMEMT 220, and the GMEMT 220 clears the valid bit of the corresponding entry, releasing the global record.

[0050] This application achieves dynamic merging of cross-thread bundle access requests for the same cache line through the aforementioned two-level missing state recording mechanism (i.e., a thread bundle-private MSHR array and a globally shared GMEMT table). Regardless of whether memory access requests from multiple thread bundles are interleaved in time, as long as the target cache line address is the same, subsequent requests are merged into the first incomplete downstream request, and the waiting thread bundle set is uniformly managed by GMEMT. After the data is returned, the broadcast engine distributes the data to all relevant thread bundles simultaneously according to the waiting set. This method fundamentally eliminates the phenomenon of repeatedly initiating downstream accesses for the same cache line, significantly reduces the bandwidth pressure on L2 cache, on-chip network, and DRAM, and reduces the repeated occupation of MSHR resources, effectively improving the memory access efficiency and system performance of GPGPU in data-intensive scenarios (such as parallel computation of large model tensors).

[0051] Figure 3 The illustration shows a schematic diagram of an example computing device in which the above-described methods and / or apparatus can be implemented according to embodiments of this application. Figure 3 As shown, computing device 300 may include bus 302 or other communication mechanism for transmitting information, and one or more hardware processors 304 coupled to bus 302 for processing information. The one or more hardware processors 304 may include, for example, one or more general-purpose microprocessors.

[0052] like Figure 3 As shown, in some embodiments, computing device 300 may further include main memory 306 coupled to bus 303. Main memory 306 is used to store information and instructions executed by one or more processors 304, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 306 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 304. When these instructions are stored in storage media accessible to one or more processors 304, they can cause computing device 300 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 308 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0053] like Figure 3As shown, in some embodiments, computing device 300 may further include one or more communication interfaces or network interfaces 310 coupled to bus 303. Network interface 310 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 310 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0054] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0055] Each of the processes, methods, and algorithms described above may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented partially or fully in dedicated circuit systems.

[0056] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0057] Specific embodiments further provide an apparatus including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the apparatus to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0058] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0059] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this application. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0060] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0061] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0062] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0063] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0064] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0065] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this application. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this application as defined by the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0066] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0067] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes can be made to these embodiments without departing from the broader scope of embodiments of this application.

[0068] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this application. Therefore, "implementation" is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their equivalents in their full scope.

Claims

1. A method for handling memory access requests in a general-purpose graphics processor, characterized in that, include: Determine whether the memory access request for the target cache line by the first thread bundle is hit in the L1 cache; In response to determining that the memory access request for the target cache line by the first thread bundle is a miss in the L1 cache, the target cache line is queried in the thread bundle missing state holding register array to see if there is a local incomplete missing record triggered by the first thread bundle. In response to the determination of a missing thread bundle state, maintain the register array to show a locally incomplete missing record and wait for the data returned from the access to the target cache line initiated to the subsequent storage level; In response to determining that there is no local incomplete missing record in the thread bundle missing state holding register array, the global missing state holding register table is queried to see if there is a global incomplete missing record for the target cache line triggered by other thread bundles; In response to the determination that there is no globally incomplete missing record for the target cache line triggered by other thread bundles in the global missing state hold register table, a new globally incomplete missing record is created for the target cache line, and an access request for the target cache line is initiated to the subsequent storage level; In response to the determination that there is a globally incomplete missing record for the target cache line triggered by other thread bundles in the global missing state holding register table, the first thread bundle is added to the waiting thread bundle set corresponding to the globally incomplete missing record for the target cache line, and waits for the return data of the access to the target cache line initiated to the subsequent storage level; after the data of the target cache line is returned, the data is distributed to all waiting thread bundles in the waiting thread bundle set through the broadcast mechanism.

2. The method according to claim 1, characterized in that, Global incomplete missing records include: the address information of the target cache line and the set of waiting thread bundles for the target cache line.

3. The method according to claim 1, characterized in that, The global missing state hold register table uses the target cache line address and memory identifier as index keys.

4. The method according to claim 1, characterized in that, The waiting thread bundle set is a thread bundle bitmap, where each bit of the thread bundle bitmap corresponds to an identifier of a thread bundle.

5. The method according to claim 1, characterized in that, Subsequent storage tiers include at least one of the following: L2 cache, on-chip network, dynamic random access memory.

6. The method according to claim 1, characterized in that, This data is simultaneously distributed to all waiting thread bundles in the waiting thread bundle set via a broadcast mechanism, including: Write this data back to the register file corresponding to all waiting thread bundles.

7. The method according to claim 1, characterized in that, The thread bundle missing state holding register array is indexed by the thread bundle identifier.

8. The method according to claim 1, characterized in that, Also includes: In response to the determination that there is a local incomplete missing record triggered by the first thread bundle in the thread bundle missing state holding register array, wait for the access return data corresponding to the local incomplete missing record, instead of querying the global missing state holding register table.

9. The method according to claim 1, characterized in that, The global missing state hold register table is located within the streaming multiprocessor of the general-purpose graphics processor.

10. A memory access request processing apparatus for a general-purpose graphics processor, characterized in that, The device includes: A thread bundle missing status holding register array is coupled to the load memory of the general-purpose graphics processor and is used to store local incomplete missing records of thread bundles for cache lines, the local incomplete missing records including the identifier of the thread bundle and the cache line requested by the thread bundle; A global missing state holding register table, coupled to the thread bundle missing state holding register array, is used to store global incomplete missing records for cache lines. The global incomplete missing record includes the address information of the cache line and the set of waiting thread bundles for that cache line; and A broadcast engine, coupled to the global missing state holding register table and the register file of the general-purpose graphics processor, is used to distribute the return data of the target cache line to all waiting thread bundles in the global incomplete missing record of that target cache line via a broadcast mechanism. The general-purpose graphics processor is configured to perform the method according to claim 1.

Images