Multi-GPU frame rendering

Abstract

Procedure comprehensive: Generating, by a first graphics processing unit, GPU (139(0)) coupled to a first memory circuit, one or more first memory access requests in connection with rendering a first primitive for a first frame, wherein at least one of the first memory access requests targets a first data page which is physically located in a second memory circuit coupled to a second GPU (139(1)); Request, by the first GPU(139(0)), the first data page via a first data link that couples the first GPU to the second GPU; Collecting, through a register circuit in the first GPU (139(0)), an access request count for the first data page; Notification of a driver by the first GPU (139(0)) that the access request count has reached a certain threshold, Received, by the first GPU (139(0)), a first copy instruction to copy the first data page from the second memory circuit via the first data link, to create a copy of the first data page in the first memory circuit before the first GPU (139(0)) accesses the first data page in connection with rendering the first primitive for a second frame, Execution of the first copy command by the first GPU (139(0)); and Generating, by the first GPU (139(0)), one or more second memory access requests in connection with rendering the first primitive, wherein at least one of the second memory access requests targets the copy of the first data page in the first memory circuit.

Description

Scope of the invention

[0001] The present invention relates to the rendering of an image and in particular to frame or single-image rendering with multiple graphics processing units. background

[0002] Two key performance indicators in a graphics rendering system are frame rate and latency. For many applications, such as augmented reality, low latency is crucial for a realistic user experience. Alternate frame rendering (AFR) can improve the frame rate by alternately assigning frames to different graphics processing units (GPUs). However, AFR typically does not improve latency and can lead to performance degradation in applications with data dependencies between frames. Split-frame rendering (SFR) is another technique that can improve the frame rate for certain applications. However, SFR is not practical for current workloads. Furthermore, both AFR and SFR impose certain computational and runtime limitations that restrict their applicability for many current graphics applications.Therefore, it is necessary to address these problems and / or other problems related to the state of the art. Reference is also made to US 2007 / 0 257 935 A1, US 8 271746 B1, US 2017 / 0 243 319 A1, US 2016 / 0 260 228 A1, and US 2014 / 0 049 549 A1. EP 2 473 915 B1 discloses a general-purpose memory with an associated processing unit. Summary

[0003] A method, a computer-readable medium, and a system according to the independent claims are disclosed for rendering graphic frames using multiple graphics processing units (GPUs). The method according to claim 1 includes, among other things, generating one or more first memory access requests by a first graphics processing unit (GPU) coupled to a first memory circuit, in conjunction with rendering a first primitive for a first frame, wherein at least one of the first memory access requests targets a first data page that is physically located in a second memory circuit coupled to a second GPU. The method further includes requesting the first data page by the first GPU via a first data link that couples the first GPU to the second GPU.Furthermore, the method includes collecting a count of access requests for the first data page by a register circuit within the first GPU. The method also includes the first GPU notifying a driver that the number of access requests has reached a certain threshold. In one embodiment, at least one of the first memory access requests targets a first data page that is physically located in a second memory circuit coupled to a second GPU.

[0004] The computer-readable medium contains instructions that, when executed by a processing unit, carry out the procedure. Furthermore, the system includes a circuit configured to execute the procedure. Brief description of the drawings Fig. According to one embodiment, 1A represents a flowchart of a method for notifying a driver that a number of requests has reached a certain threshold; Fig. According to one embodiment, 1B represents a flowchart of a method for accessing a copied data page; Fig. According to one embodiment, Figure 1C represents a flowchart of a method for controlling the execution of a copy command for a data page; Fig. According to one embodiment, 1D represents a method for assigning rendering tasks depending on a screenspace checkerboard pattern; Fig. According to one embodiment, 1E represents a system for transmitting compressed data over a high-speed data connection; Fig. According to one embodiment, 1F represents a system with hardware counters for collecting access request counts within a graphics processing unit; Fig. According to one embodiment, 1G represents an exemplary sequence for copying and accessing data pages; Fig. 2 represents a parallel processing unit according to one embodiment; Fig. 3A represents a general processing cluster of the parallel processing unit of Fig. 2 according to one embodiment; Fig. 3B represents a partition unit of the parallel processing unit of Fig. 2 according to one embodiment; Fig. 4 represents the streaming multiprocessor of Fig. 3A according to one embodiment; Fig. Figure 5 represents an exemplary system in which the different architectures and / or functionalities of the various previous embodiments can be implemented; Fig. Figure 6 is a conceptual representation of a graphics processing pipeline, which is fed by the parallel processing unit of Fig. 2 is implemented according to an embodiment; Fig. Figure 7A is a conceptual representation of a configuration with two processors according to one embodiment; Fig. Figure 7B is a conceptual representation of a four-processor configuration according to one embodiment; and Fig. 7C is a conceptual representation of a second configuration with four processors according to one embodiment. Detailed description

[0005] As described below, distributing rendering across multiple GPUs reduces rendering latency and provides a more general and scalable GPU computing environment compared to traditional techniques. Specifically, multi-GPU rendering provides a more general and scalable GPU computing environment than traditional split-frame rendering (SFR) or alternative-frame rendering (AFR) methods.

[0006] In one embodiment, two or more GPUs are configured to operate as peers, with one peer connected via a high-speed data link (e.g., NVLINK, high-speed data link 150). Fig. 1E) can access data (e.g., surfaces) in the local memory of another peer. For example, a first GPU of two or more GPUs can perform structural mapping using surface data located remotely in the memory of a second GPU of the two or more GPUs. Based on tracking data of the remote memory access, the remote surface data can be copied to the local memory of the first GPU, since the bandwidth and / or latency of the high-speed data link may be worse than that of a local memory interface. In certain embodiments, a frame to be rendered is divided into regions (e.g., rectangular regions) that form a checkerboard pattern, with non-overlapping adjacent regions sharing a common border in the checkerboard pattern, which are generally assigned to different GPUs.In other embodiments, the frame is divided into regions that can overlap by one or more pixels (e.g., to find a compromise between redundant calculations and potentially reduced interprocess communication). For large surfaces, the regions are aligned with the memory page boundaries for render targets and distributed textures or structures. In one embodiment, the number of regions is dynamically determined and updated for new frames to reduce long-distance transmissions and enable overall load balancing between the two or more GPUs. The rectangles are rendered separately by the different GPUs for the frame and combined into a complete frame in a frame buffer. In one embodiment, the frame buffer resides in the local memory of one of the two or more GPUs.

[0007] In certain embodiments, primitives (e.g., represented by primitive instructions) are passed to the two or more GPUs for rendering the frame, and hardware circuitry within them provides clipping windows for selecting which primitives are contained within a given chessboard rectangle. Unselected primitives are discarded early to reduce GPU resource utilization. In one embodiment, the discarding of an entire primitive is accelerated by a primitive engine, which checks a given triangle-bounding field for an intersection with a currently actively owned area (e.g., a rectangle in the chessboard pattern assigned to a given GPU). Such primitive discarding can be performed by hardware logic circuitry, which may be placed after a positional transformation stage in a rendering pipeline.In one embodiment, the transformed attributes of a discarded triangle are not written to memory, and the triangle is not rasterized, thus saving both processing cycles and memory bandwidth. In another embodiment, pixel-precise discarding is implemented in a raster circuit or in a raster shader function.

[0008] Each of the two or more GPUs renders selected primitives within assigned regions. Rendering can involve multiple render passes, and the results of a render pass, stored on one or more surfaces, can be used by the two or more GPUs for one or more subsequent render passes. Rendering a particular pass for an assigned region on a first GPU may require remote data from a second GPU. The remote data can be fetched from the second (remote) GPU on demand in response to a request from the first GPU, or the remote data can be copied asynchronously in advance of an anticipated request, potentially achieving lower overall latency.In many common scenarios, sequentially rendered frames are self-similar, and a memory access request pattern for one frame is essentially replicated in a subsequent frame, making each frame a good predictor of access patterns in a subsequent frame. For example, two sequential frames may contain essentially the same instruction primitives, with each generating essentially the same memory access requests as it is rendered. Consequently, an access pattern observed in conjunction with the execution of a first instruction stream to render a first frame can be used to predict which memory blocks should be copied before the execution of a second instruction stream to render a second frame.In certain embodiments, data within memory pages is stored in a compressed format, and remote data is copied in the compressed format to reduce the overall utilization of the high-speed data connection.

[0009] Fig. Figure 1A, according to one embodiment, represents a flowchart of a method 100 for notifying a driver that a request counter or a number of requests has reached a certain threshold. Although the method 100, in conjunction with the systems of Fig. As described in Sections 2-7, any processing system implementing Method 100 is within the scope and spirit of the embodiments described in this disclosure. In various embodiments, Method 100 is implemented in the context of a graphics system configured to render graphics frames from image information comprising graphics primitives. One or more operations of Method 100 can be performed by task-specific circuitry or by a combination of task-specific circuitry and general-purpose processing units. In one embodiment, Method 100 is performed by a processing system that includes a general-purpose central processing unit (CPU) and a parallel processing unit (PPU), such as the PPU 200 described in [reference missing]. Fig. 2, or any combination thereof.

[0010] In one embodiment, the processing system includes a first GPU directly coupled to a first memory circuit and a second GPU directly coupled to a second memory circuit. Furthermore, the first GPU is coupled to the second GPU via the high-speed data link. In one embodiment, the high-speed data link provides atomic peer access operations and transfers data at a rate of at least one gigabyte per second. The first memory circuit is not directly coupled to the second GPU, and the second memory circuit is not directly coupled to the first GPU.

[0011] In step 102, the first GPU generates one or more initial memory access requests in conjunction with rendering an initial primitive (e.g., when executing an initial primitive instruction) of the initial instruction stream, where at least one of the initial memory access requests targets an initial data page that is physically located within the second memory circuit. In this context, the initial primitive is associated with an initial frame.

[0012] In step 104, the first GPU requests the first data page over the high-speed data link. This request can be a read access request without restriction. To predict which data pages should be copied in advance from remote memory to a first memory that is local to the first GPU, accesses to remote memory are tracked within each GPU. Specifically, hardware counters are configured to count access requests to different data pages in memory and report high-frequency pages to a GPU driver. The hardware counters (registers) can be configured to selectively (by incrementing) collect access requests to different remote pages while the frame (the current frame) is being rendered.For example, the hardware counters can be configured to selectively collect access requests only for specific contexts or subcontexts of a particular render run. A specific counter can be restricted to collecting access requests for a particular render run and can be reset at the beginning of the render run. In one embodiment, the hardware counters each have a programmable address range for collecting a number of access requests. Only an access request with an address within the programmable address range can cause a specific hardware counter to be incremented.

[0013] In step 106, a register circuit (e.g., a hardware counter) collects a number of access requests for the first data page. For example, each access to the first data page can cause the register circuit to increment an accumulated count by one. In one embodiment, the count is initialized with an integer value of N and decremented by one with each access. In another embodiment, the count is initialized (e.g., with zero) and incremented until a programmed threshold is reached.

[0014] In one implementation, if the number of accesses to a specific remote page exceeds a certain threshold, the GPU driver is notified and an identifier for the page (e.g., address and render pass) is added to a collection of high-frequency pages. The GPU driver can be configured to schedule high-frequency pages to be copied from the local memory of one GPU (e.g., using a hardware copy engine), which writes the pages to the local memory of a second GPU that reads them. Because the high-frequency pages are copied to the local memory of the second GPU, remote traffic can be reduced. By separating the access counts according to the render pass, the GPU driver can better plan when to copy a specific page within the overall render sequence for a given frame.Furthermore, hardware counters can be allocated more efficiently by restricting which contexts are counted.

[0015] At step 108, the first GPU notifies a driver that the number of access requests has reached a certain threshold. The driver can be a software driver designed to run at any technically feasible location within a given system architecture. For example, the driver can run within a central processing unit (CPU) responsible for managing the operation of the first and second GPUs.

[0016] In one embodiment, the first primitive is rendered by the first GPU according to a first primitive instruction contained in a first instruction stream for a first frame, and the second frame is rendered after the first frame. In another embodiment, the first instruction stream specifies a first render pass that is performed in conjunction with the rendering of the first frame, and a second instruction stream specifies the same render pass that is subsequently performed in conjunction with the rendering of the second frame, and notification occurs during the rendering of the first frame.

[0017] Further illustrative information on various optional architectures and features is now presented, which may or may not allow the aforementioned framework to be configured according to the user's wishes. It should be expressly noted that the following information is provided for illustrative purposes and is not to be interpreted as restrictive. One of the following features may be present optionally, with or without the exception of other described features.

[0018] Fig. Figure 1B, according to one embodiment, represents a flowchart of a method 110 for accessing a page containing copied data or a copied data page. Although the method 110, in conjunction with the systems of Fig. As described in Sections 2-7, any processing system implementing Method 110 is within the scope and spirit of the embodiments described in this disclosure. In various embodiments, Method 110 is implemented in the context of a graphics system configured to render graphics frames from image information comprising graphics primitives. One or more operations of Method 110 can be performed by task-specific circuits or by a combination of task-specific circuits and general-purpose processing units. In one embodiment, Method 110 is performed by a processing system that includes a general-purpose central processing unit (CPU) and a parallel processing unit (PPU), such as the PPU 200 described in [reference missing]. Fig. 2, or any combination thereof. In one embodiment, method 110 is executed in response to receiving a copy command from the driver. Furthermore, the driver schedules the copy command upon receiving a notification, as described in step 108 of method 100.

[0019] In step 112, the first GPU receives a first copy instruction to copy the first data page from the second memory circuit over the high-speed data link to create a copy of the first data page within the first memory circuit. In one embodiment, the first copy instruction is executed before the first GPU accesses the first data page in conjunction with rendering the first primitive. In another embodiment, the first primitive is rendered by the first GPU according to a first primitive instruction contained in a first instruction stream for a first frame, and the second frame is rendered after the first frame. Rendering the first primitive for the second frame may cause the first GPU to generate one or more second memory access requests for data located in the copy of the first data page residing in the first memory circuit.

[0020] In step 114, the first GPU executes the first copy instruction to copy the first data page from the second memory circuit (a local memory of the second GPU) to the first memory circuit (a local memory of the first GPU). In step 116, the first GPU generates the one or more second memory access requests targeting the first data page located in the first memory circuit, in conjunction with rendering the first primitive for a second frame.

[0021] Fig. Figure 1C, according to one embodiment, represents a flowchart of a method 120 for controlling the execution of a copy command for a data page. Although the method 120, in conjunction with the systems of Fig. As described in Sections 2-7, any processing system implementing Method 120 is within the scope and spirit of the embodiments described in this disclosure. In various embodiments, Method 120 is implemented in the context of a graphics system configured to render graphics frames from image information comprising graphics primitives. One or more operations of Method 120 can be performed by task-specific circuits or by a combination of task-specific circuits and general-purpose processing units. In one embodiment, Method 120 is performed by a processing system that includes a general-purpose central processing unit (CPU) and a parallel processing unit (PPU), such as the PPU 200 described in [reference missing]. Fig. 2, or any combination thereof. In one embodiment, a driver executed within the CPU is configured to execute method 120.

[0022] At step 122, the driver receives a notification that the number of access requests for the first data page has reached a certain threshold. For example, during the rendering of the first frame, the first GPU can access the first data page located in the second memory circuit of the second GPU, as described in step 104 of Fig. 1A is described. If the first GPU accesses the first data page multiple times, corresponding to a number equal to the specified threshold, the first GPU notifies the driver, causing it to receive the notification as described in step 108 of Fig. As described in 1A. Alternatively, if the second GPU has a number of access requests corresponding to the threshold on the first data page of the first GPU, it can notify the driver, thus enabling the driver to receive the notification.

[0023] In step 124, the driver schedules a copy instruction to copy the first data page from the second GPU to the first GPU. In practice, the first data page might reside within the second GPU's second memory circuit or within a cache of the second GPU. The first data page is transferred to the first GPU and stored as a copy in the first GPU's first memory circuit or in a cache of the first GPU. In step 126, the driver specifies the copy instruction within a GPU instruction stream for rendering the second frame. In one embodiment, the driver inserts the copy instruction into the GPU instruction stream.

[0024] In one embodiment, the GPU instruction stream is the second instruction stream for the first GPU, and the copy instruction can cause the first GPU to perform a copy operation specified by the copy instruction. In an alternative embodiment, the GPU instruction stream is an instruction stream for the second GPU, and the copy instruction can cause the second GPU to perform a copy operation specified by the copy instruction.

[0025] In one embodiment, the copy instruction is contained in a special copy instruction stream, while graphics primitives are contained in a general instruction stream. In such an embodiment, a semaphore can be used to mediate and synchronize the progress of the copy instruction stream and the general instruction stream. For example, the semaphore mechanism can be configured to ensure the completion of the copy instruction before the start of a particular rendering pass that includes instructions from the general instruction stream accessing the first data page. In addition to a rendering pass acting as a synchronization barrier, individual graphics primitives and / or general instructions can also serve this purpose, and the execution of specific instructions can be mediated by a semaphore.

[0026] In another embodiment, the copy instruction and the graphics primitives are integrated into a unified instruction stream. A given copy instruction can be executed asynchronously to other instructions in the unified instruction stream, and any technically feasible method (including a semaphore per copy instruction) can be implemented to create a synchronization barrier between the completion of the otherwise asynchronous copying and the execution of instructions that depend on copied data.

[0027] During normal operation, the driver can receive notifications from the first GPU, the second GPU, and optionally from additional GPUs. In a typical operating mode, the first GPU writes data to a specific data page (e.g., during an initial render pass), and the first GPU then references the data multiple times during subsequent processing (e.g., in a later render pass). The notifications identify certain pages as high-traffic (hot) pages that can benefit from being copied to local storage, where accesses occur locally rather than repeatedly over the high-speed data connection. Using the information from the notification, the driver can then issue a copy command for a high-traffic data page (e.g.,The driver can schedule the first data page (determined during the rendering of the first frame) so that this command is executed before the high-frequency access over the high-speed data connection associated with rendering a second frame. For example, the driver can schedule a copy command to copy the first data page from the second memory circuit to the first memory circuit, depending on the access intensity to the first data page while a frame is being rendered, before the first GPU needs to access the first data page to render a subsequent frame. In this way, the driver is able to adaptively manage data location to improve overall performance.

[0028] The allocation of rendering tasks between the first and second GPUs can be performed using any technically feasible method. One such method or technique is described in... Fig. 1D representation. In general, the rendering work can be divided between two or more GPUs using any technically feasible method, without losing the scope and spirit of the different implementations.

[0029] Fig. 1D presents a method for assigning rendering work based on a screenspace checkerboard pattern according to one embodiment. As shown, a screenspace image or screen area 130 is divided into rectangular areas 132. Different rectangular areas 132 can be assigned to different GPUs for rendering. In one embodiment, the rectangular areas 132 are assigned to one of two different GPUs according to a checkerboard pattern covering the screen area 130. For example, rectangular areas 132(1), 132(3), 132(5), and 132(7) can be assigned to one GPU 139(0) (e.g., the first GPU), and rectangular areas 132(0), 132(2), 132(4), 132(6), and 132(8) can be assigned to one GPU 139(1) (e.g., the second GPU). The screen area of ​​130 can handle primitives (e.g.geometric primitives) such as triangles 134, each of which may be entirely contained within a rectangular area 132, or which may span one or more rectangular areas 132. For example, triangle 134(0) is entirely contained within rectangular area 132(0), while triangle 134(2) is contained within rectangular area 132(3) and rectangular area 132(4). In alternative embodiments, the rectangular areas 132 may be assigned to three or more different GPUs according to a different pattern.

[0030] In one embodiment, the GPU 139(0) includes a clip map 131(0) that specifies which rectangular areas 132 are to be processed by the GPU 139(0). Likewise, the GPU 139(1) includes a clip map 131(1) that specifies which areas 132 are to be processed by the GPU 139(1). Areas to be processed by the GPU 139(0) are shaded in clip map 131(0), while areas to be processed by the GPU 139(1) are shaded in clip map 131(1). In this example, the union of clip map 131(0) and clip map 131(1) completely covers the screen area 130. In another embodiment, a clip map 131(0) is stored identically in GPU 139(0) and GPU 139(1), but GPU 139(1) has an inverted bit set, which causes GPU 139(1) to use an inverted representation of the clip map 131(0). In yet another embodiment, clip maps 131 are programmed orThe GPUs 139 are determined by program. For example, the rectangular areas 132 that are assigned to a particular GPU for rendering (i.e., are "owned" by the GPU) can be derived using a formula that works with and / or adapts a tile coordinate space.

[0031] Geometric primitives, such as triangles 134, can be transferred to both GPU 139(0) and GPU 139(1), wherein a corresponding clipping circuit 138 is configured to either retain or discard a specific geometric primitive, depending on whether the geometric primitive intersects a rectangular area 132 allocated for processing by a corresponding GPU. In one embodiment, a clipping circuit 138(0) can be configured in GPU 139(0) to discard geometric primitives that do not cover or intersect a rectangular area to be processed by GPU 139(0). Likewise, a clipping circuit 138(1) can be configured in GPU 139(1) to discard geometric primitives that do not cover or intersect a rectangular area to be processed by GPU 139(1). The clipping circuit 138 can perform all necessary transformation operations (e.g.to perform primitive vertices) to map geometric primitives from any space (e.g., worldspace) to screenspace before performing clipping or truncation operations in screenspace.

[0032] By discarding entire geometric primitives early on, memory and computational resources are advantageously conserved, and each GPU 139 can operate more efficiently. Detailed pixel or subpixel coverage of geometric primitives can be determined during a rasterization operation on a specific geometric primitive later in a render pipeline for that primitive.

[0033] In certain embodiments, the overall compute traffic is distributed such that cooperative thread groups (CTAs, or blocks in Microsoft™ DirectX terminology) are halved, with one half of the threads executing on GPU 139(0) and the other half on GPU 139(1). The CTA splitting can be achieved by pre-executed shader code, resulting in the early discarding of half of the CTAs on each GPU 139. In one embodiment, the clipping circuit 138 is configured to discard CTAs.

[0034] In one embodiment, a shader compiler is configured to include instructions in a CTA that perform a test for critical surface access at the beginning of a CTA shader program. Hardware support allows the instructions to determine whether the requested data associated with the access belongs to the local GPU. If most threads in the CTA return false, the CTA is dropped. It should be noted that the remote GPU performs the same test on an equivalent CTA, thus ensuring that the CTA on the remote GPU survives. In light of the present disclosure, the person skilled in the art understands that this method optimizes the CTA distribution for the chosen memory access instruction(s) with respect to memory locations.

[0035] A workload 136(0) has rectangular regions 132 allocated to GPU 139(0). Geometric primitives within rectangular regions of workload 136(0) are selected by clipping circuit 138(0) to be included in workload 136(0). Clipping circuit 138(0) discards geometric primitives that are not contained within a rectangular region 132 allocated to GPU 139(0). Similarly, a workload 136(1) contains rectangular regions 132 allocated to GPU 139(1). Geometric primitives to be processed by a particular GPU are stored in memory for that GPU, while other geometric primitives (e.g., those to be processed by a different GPU) are discarded by the GPU, thus saving memory bandwidth and storage space. The geometric primitives discarded by one GPU are stored and processed by another GPU.In cases where a geometric primitive covers two or more rectangular areas assigned to two or more GPUs, the geometric primitive is instantiated in each covered rectangular area, and different sections of the geometric primitive can be rendered by a suitable GPU. For example, triangle 134(1) within rectangular area 132(1) can be rendered by GPU 139(0), triangle 134(0) within rectangular area 132(0) can be rendered by GPU 139(1), and at least one first section of triangle 134(2) can be rendered by GPU 139(0), while at least one second section of triangle 134(2) can be rendered by GPU 139(1).

[0036] In one embodiment, the rendering results of the rectangular areas 132 are combined in a single GPU to produce a final rendered frame. In various embodiments, any number of GPUs can work together to produce a final rendered frame. In certain embodiments, two or more final rendered frames are produced. For example, left and right frames can be generated to be displayed to the respective left and right eyes of a user with a virtual reality headset. In another example, multiple monitors are configured to display different rendered frames, e.g., from different camera positions (e.g., to create a panoramic view of an image).

[0037] Fig. 1E represents a system 140 for transmitting compressed data over a high-speed data link 150 according to one embodiment. The system 140 includes the GPU 139(0), the GPU 139(1), and the high-speed data link 150. The system 140 may also include other subsystems not shown here. The high-speed data link 150 is configured to transmit data between a client 148 within the GPU 139(0) and a client 148 within the GPU 139(1). For example, the client circuit 148(A) within the GPU 139(0) can transmit compressed data via an internal link 152 to the crossbar 146(0) and the high-speed hub 142(0). The compressed data can be transferred in a compressed format via the high-speed connection 150 to a decompression circuit 144 within the HSBUB 142(1) of the GPU 139(1).The decompression circuit 144 is configured to decompress data transmitted in compressed format to generate decompressed data. The decompressed data can be transmitted via the internal connection 153 to the coupling array 146(1), which can forward the decompressed data to the internal connection 154 and finally to a client circuit 148(D). In one embodiment, the high-speed data link 150 comprises a commercial high-speed link known in the prior art as NVLink. In certain embodiments, the high-speed data link 150 is configured to transmit compressed data, thereby reducing the transmission time and the overall utilization of the high-speed data link 150. The client circuits 148 can include any technically feasible subsystems within a GPU 139.For example, a client circuit 148 can have a memory interface to local memory, a cache, etc. In one embodiment, a compression block (not shown) can be arranged and configured along the internal connection 154 to compress and / or decompress data passing through it. The compression block can provide appropriately structured data for the XBAR 146 to allocate it to different memory sections (e.g., which may differ with respect to source and destination addresses).

[0038] Fig. According to one embodiment, 1F represents a system 160 that includes hardware counters 171, 172 for summing the number of access requests within a graphics processing unit (e.g., GPU 139). As shown, the system 160 includes a GPU 139 and DRAMs 168 configured as local memory directly coupled to the GPU 139. In one embodiment, the system 160 includes one or more CPUs / peer processing units 178, 179.

[0039] The GPU 139 includes hub clients 162, a frame buffer hub (FBHUB) 163, an HSHUB 142, at least one coupling array (XBAR) 165, one or more general-purpose processing clusters (GPCs) 166, one or more level 2 (L2) caches 167, and an access counter subsystem 170 comprising a number C of hardware counters 171, 172. Furthermore, the GPU 139 may also include a ring buffer memory 174 configured to store a number D of buffer entries 175. In one embodiment, the L2 caches 167 are configured to hold a cached copy of certain data stored in DRAMs 168 and / or in other memory circuits.

[0040] The DRAMs 168 are designed to provide reading / writing of local data, such as programming instructions, image data, structure data, and other general data. The Hub Clients 162 can include specialized processing units (e.g., a primitive distributor or "PD") and other circuitry designed to process data stored in the DRAMs 168 or L2 caches 167 and / or data stored in memory circuitry coupled to the FBHUB 163 and / or the HSHUB 142. In one embodiment, the FBHUB 163 and / or the HSHUB 142 are configured to provide access-specific operations related to data transmission between a system interface 161 (e.g. PCIe), the high-speed data connection 150 and internal connections coupled to the subsystem 170 for counting accesses, the hub clients 162 and the at least one coupling field 165.In one embodiment, the GPCs 166 comprise the GPCs 250 from . Fig. 2, and at least one coupling field 165 includes coupling field 270. In addition, the L2 caches 167 can be used as L2 cache 360 ​​of the Fig. 3B must be implemented. At least one coupling field 165 can be a coupling field 146. Fig. exhibit 1E.

[0041] The CPUs / peer processing units 178, 179 can include one or more GPUs and / or one or more CPUs. The GPUs can be configured to operate as peers or equivalent computing units. In one embodiment, peer processing units are configured to provide at least one other peer processing unit with access to locally attached memory (e.g., DRAMs 168). A particular processing unit can configure a memory area to have local memory in one area of ​​the memory area and peer memory as an access point or aperture in another area of ​​the memory area. When a memory request is made within the processing unit, the memory request is directed either to local memory or to a peer processing unit, depending on an address range or base address for the memory request.

[0042] Access to the memory of working shaders can be achieved via a generic memory routing mechanism, where a memory management unit (not shown) determines whether a requested page is located remotely (peer) or locally. An access request to the requested page is forwarded accordingly as a local or remote access. In one embodiment, a peer access ("peer aperture") defines a memory area mapped to a peer GPU. Data stored within the peer access can be cached in a local cache, and atomic operations of a peer can be performed in the peer access memory area via the high-speed data link 150. This feature contrasts with industry-standard interconnects such as PCI Express (PCIe), which generally do not support the combination of atomic operations of an access and a peer.

[0043] In one embodiment, the access counting subsystem 170 comprises hardware counters 171 and 172 and a control circuit 173. Each hardware counter 171 and 172 can contain status information. This status information can include, without limitation, the current number of requests, a threshold, a processing pass, a channel mask, a context, and an address range. Furthermore, the hardware counters 171 and 172 can be configured to operate with virtual or physical addresses. The access counting subsystem 170 is configured to provide visibility into all access requests that pass through at least the HSHUB 142 and the FBHUB 163. This visibility can include, without limitation, an address, a render pass identifier, a context identifier, and a subcontext identifier.In one embodiment, each access request can be made to either a physical address or a virtual address, and the visibility has an operating mode setting with respect to physical or virtual requests.

[0044] If a requested address falls within a specific address range specified for a particular hardware counter (171, 172), and other conditions for the hardware counter are met, the current request count for the hardware counter is incremented. These other conditions can include, without limitation, that a current processing pass matches a processing pass specified in the state information, a current context matches a processing context specified in the state information, and a current subcontext matches at least one bit representing a true state in the channel mask.

[0045] A subcontext can be specified and assigned to a particular instance of a thread running within GPU 139. The channel mask can contain any number of bits (e.g., 64 bits) that can be set or left unset to include or exclude specific threads from being counted by certain hardware counters 171, 172.

[0046] In one embodiment, a specific hardware counter 171 is compared with an associated hardware counter 172. Under normal operation, hardware counter 171 and hardware counter 172 can be incremented when necessary and identical conditions are met, including a matching pass ID and address range. However, when a different (new) pass ID is received, hardware counter 171 can reset the current request count. The current request count is not reset on the associated hardware counter 172 when a new pass ID is received. In this way, data relevant to a particular pass can be identified for preloading before that pass, while "hot" address pages can be monitored separately and continuously and identified as potential candidates for migration from one GPU to a peer GPU.

[0047] When the current number of requests reaches the threshold, subsystem 170 notifies a driver for access counting. The notification can include a counter identifier or other identifying information, such as an address range, page number, or page number range. The threshold can be constant or programmable for each hardware counter 171 and / or 172.

[0048] In one embodiment, a hardware counter 171 is assigned from a pool of hardware counters (171(0) to 171(C)) when a new address range occurs. An assigned or associated hardware counter 172 can also be assigned together with the hardware counter 171. If no free hardware counters are available from the pool, a currently assigned hardware counter 171 is released according to a replacement policy. An example of a replacement policy is a "longest unused" policy, in which a hardware counter that has not been accessed for the longest time is deleted and assigned to count the new address range. Other technically feasible replacement policies can also be used.

[0049] A ring buffer memory 174 can be configured to receive notifications from the hardware counters 171, 172 and queue the notifications as buffer entries 175, which are then sent to the DRAMs 168 and / or further to the driver. In one embodiment, the driver is configured to run on a CPU (e.g., the CPUs / peer processing units 178, 179). The ring buffer memory 174 can include a control circuit 176 configured to manage received notifications, generate buffer entries 175 from the notifications, and transfer the notifications to DRAM 168 and / or the driver.

[0050] Fig. Figure 1G represents an exemplary sequence 180 for copying and accessing data pages according to one embodiment. The sequence 180 represents a sequence of frames 182(0), 182(1), etc., along a time axis 181. The rendering of each frame 182 includes a rendering pass 184, which comprises a sequence of instructions (e.g., primitive instructions) in an instruction stream. As shown, during the rendering pass 184(0) at frame 182(0), a threshold number of access requests is generated from a first GPU 139 to the same data page in a memory of a second GPU 139, thereby generating a notification 186. The access requests are counted by a hardware counter 171. It may be necessary for the access requests to meet certain criteria in addition to a specific address range. For example, access requests may need to be processed during a specific render run (e.g., during a specific rendering cycle).B. rendering pass 184) of a specific frame. Furthermore, the access requests may need to occur during the execution of a specific context. Additionally, the access requests may need to be generated by threads with a subcontext selected by a channel mask for the hardware counter 171. These requests can be used by the driver to schedule when a copy operation 187 should be performed.

[0051] Notification 186 indicates that a number of accesses to an identified page (or pages) of peer storage data has reached the access threshold, and that the data page (or pages) should be copied before render pass 184 is executed on a subsequent frame 182. For example, notification 186 is generated during render pass 184(0) of frame 182(0). Therefore, the driver schedules a copy operation 187 to occur before render pass 184(1) of frame 182(1). A semaphore may be implemented to enforce the order between the completion of copy operation 187 and the start of render pass 184(1). Alternatively, the semaphore can enforce the sequence between the completion of the copy operation 187 and the execution of the first command of the render pass 184(1) that accesses the identified side of the data.After completion of the copying process 187, the data page can be accessed by the first GPU 139 from a local copy, thereby advantageously reducing the latency and reducing the utilization of the high-speed connection 150.

[0052] In one embodiment, the threshold is calculated to essentially correspond to the amount of data associated with copying one (or more) pages. The threshold can be calculated based on a specific number of bytes, on whole pages, or on a utilization factor for the high-speed 150 line. In one embodiment, the threshold is calculated by the driver. In another embodiment, the threshold is calculated according to a predefined system design.

[0053] In an exemplary embodiment, the first GPU 139 receives and executes a copy command to perform the copy operation 187 with respect to the identified data page from the memory of the second GPU 139, before the first GPU 139 accesses the identified data page in conjunction with the rendering pass 184(1) of frame 182(1). Parallel processing architecture

[0054] Fig. Figure 2 represents a parallel processing unit (PPU) 200 according to one embodiment. In one embodiment, the PPU 200 is a multithreaded processor implemented on one or more integrated circuit devices. The PPU 200 is a latency-hiding architecture designed to process a large number of threads in parallel. A thread (i.e., an execution thread) is an instantiation of a set of instructions configured for execution by the PPU 200. In one embodiment, the PPU 200 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) image data to generate two-dimensional (2D) image data for display on a display device, such as a liquid crystal display (LCD). In other embodiments, the PPU 200 can be used to perform general-purpose computations.Although an exemplary parallel processor is shown herein for illustrative purposes, it should be noted that this processor is shown for illustrative purposes only and that any processor can be used as a supplement and / or replacement for it.

[0055] As in Fig. As shown in Figure 2, the PPU 200 includes an input / output (I / O) unit 205, a host interface unit 210, a front-end unit 215, a compute scheduler unit (CSU) 220, a compute work distribution unit (CWDU) 225, a graphic primitive distribution unit (GPDU) 230, a hub 235, a crossbar (Xbar) 270, one or more general processing clusters (GPCs) 250, and one or more memory partition units 280. The PPU 200 can be connected to a host processor or other peripheral devices via a system bus 202. The PPU 200 can also be connected to local memory comprising a number of memory devices 204. In one embodiment, the local memory can comprise a number of dynamic random access memory (DRAM) devices.

[0056] The I / O unit 205 is configured to send and receive communications (i.e., commands, data, etc.) from a host processor (not shown) via the system bus 202. The I / O unit 205 can communicate with the host processor directly via the system bus 202 or via one or more intermediaries, such as a memory bridge. In one embodiment, the I / O unit 205 implements a Peripheral Component Interconnect Express (PCIe) interface for communication over a PCIe bus. In alternative embodiments, the I / O unit 205 can implement other types of known interfaces for communication with external devices.

[0057] The I / O unit 205 is coupled to a host interface unit 210, which decodes packets received via the system bus 202. In one embodiment, the packets represent commands configured to cause the PPU 200 to perform various operations. The host interface unit 210 transmits the decoded commands to various other units of the PPU 200, as specified in the commands. For example, some commands may be transmitted to the front-end unit 215. Other commands may be transmitted to the hub 235 or other units of the PPU 200, such as one or more copiers, a video encoder, a video decoder, a power management unit, etc. (not explicitly shown). In other words, the host interface unit 210 is configured to manage communication between and among the various logical units of the PPU 200.

[0058] In one embodiment, a program executed by the host processor encodes an instruction stream in a buffer, which provides the PPU 200 with work packages for processing. A work package can comprise a set of instructions and references to data to be processed by those instructions. The buffer is a region in memory accessible (i.e., read / write) to both the host processor and the PPU 200. For example, the host interface unit 210 can be configured to access the buffer in system memory connected to the system bus 202 via memory requests transmitted by the I / O unit 205 over the system bus 202. In one embodiment, the host processor writes the instruction stream to the buffer and then sends a pointer to the beginning of the instruction stream to the PPU 200.The host interface unit 210 manages the flow control of instructions from one or more command streams written by the host processor (i.e., channels) to the various subunits of the PPU 200.

[0059] The front-end unit 215 receives instructions from the host interface unit 210 from one or more command streams and forwards these instructions to the appropriate subunit of the PPU 200. Instructions associated with a compute pipeline can be received by the front-end unit 215. These compute instructions are then forwarded to a compute flow control unit 220. The compute flow control unit 220 is configured to track state information related to the various tasks it manages. The state can indicate which GPC 250 a task is assigned to, whether the task is active or inactive, the priority level assigned to the task, and so on. The compute flow control unit 220 manages the execution of a variety of tasks on one or more GPCs 250.

[0060] The computational control unit 220 is coupled to a computational workload distribution unit 225, which is configured to distribute tasks for execution across the GPCs 250. The workload distribution unit 225 can track a number of scheduled tasks received from the computational control unit 220. In one embodiment, the workload distribution unit 225 maintains a pool of pending tasks and a pool of active tasks for each of the GPCs 250. The pool of pending tasks can contain a number of slots (e.g., 16 slots) for tasks scheduled to be processed by a specific GPC 250. The pool of active tasks can contain a number of slots (e.g., 4 slots) for tasks that are actively being processed by the GPCs 250.When a GPC 250 finishes executing a task, that task is removed from the active task pool for the GPC 250, and one of the other tasks from the pending task pool is selected and scheduled for execution on the GPC 250. If an active task on the GPC 250 was idle, for example, while waiting for a data dependency to be resolved, the active task can be removed from the GPC 250 and returned to the pending task pool, while another task from the pending task pool is selected and scheduled for execution on the GPC 250.

[0061] Returning to the frontend unit 215, instructions associated with a graphics pipeline can be received by the frontend unit 215. These graphics instructions are then forwarded to a graphics primitive distribution unit 230. The graphics primitive distribution unit 230 retrieves vertex data from memory 204 or system memory via the system bus 202 for various graphics primitives. Graphics primitives can include points, lines, triangles, quads, triangular strips, and the like. The graphics primitive distribution unit 230 groups the vertices into groups of primitives and assigns tasks to the GPCs 250 to process the groups of primitives. This processing can involve executing a shader (i.e., a vertex shader, tessellation shader, geometry shader, etc.).) on a programmable processing unit as well as the execution of defined-function operations on the vertices, such as clipping, culling and viewport transformation, by means of a defined-function unit.

[0062] The Computing Work Distribution Unit 225 and the Graphics Primitive Distribution Unit 230 communicate with one or more GPCs 250 via an XBar 270. The XBar 270 is a network that connects many of the PPU 200 units to other PPU 200 units. For example, the XBar 270 can be configured to connect the Computing Work Distribution Unit 225 to a specific GPC 250. Although not explicitly shown, one or more other PPU 200 units are connected to the host unit 210. These other units can also be connected to the XBar 270 via a Hub 235.

[0063] Tasks associated with the compute pipeline are managed by the compute flow control unit 220 and distributed to a GPC 250 by the compute workload distribution unit 225. Tasks assigned to the graphics pipeline are managed by the graphics primitive distribution unit 230 and distributed to a GPC 250. The GPC 250 is configured to process the tasks and generate results. The results can be taken over by other tasks within the GPC 250, forwarded to another GPC 250 via the XBar 270, or stored in memory 204. The results can be written to memory 204 by the memory partition units 280, which provide a memory interface for reading and writing data to / from memory 204.In one embodiment, the PPU 200 includes a number U of storage partition units 280, which is equal to the number of separate and distinct storage devices 204 coupled to the PPU 200. A storage partition unit 280 is referred to below in connection with the . Fig. 3B is described in more detail.

[0064] In one embodiment, a host processor executes a driver kernel that implements an application programming interface (API). This API allows one or more applications running on the host processor to schedule operations for execution on the PPU 200. An application can generate instructions (e.g., API calls) that cause the driver kernel to create one or more tasks for execution by the PPU 200. The driver kernel outputs tasks to one or more streams, which are processed by the PPU 200. Each task can comprise one or more groups.

[0065] Fig. 3A presents a GPC 250 of the PPU 200 from Fig. 2 according to one embodiment. As in Fig. As shown in Figure 3A, each GPC 250 includes a number of hardware units for processing tasks. In one embodiment, each GPC 250 includes a pipeline manager 310, a pre-raster operations unit (PROP) 315, a raster engine 325, a work distribution crossbar (WDX) 380, a memory management unit (MMU) 390, and one or more texture processing clusters (TPCs) 320. It should be noted that the GPC 250 is Fig. 3A instead of or in addition to the ones in Fig. The units shown in 3A may contain additional hardware units.

[0066] In one embodiment, the operation of the GPC 250 is controlled by the pipeline manager 310. The pipeline manager 310 manages the configuration of one or more TPCs 320 for processing tasks assigned to the GPC 250. In one embodiment, the pipeline manager 310 can configure at least one of the TPCs 320 to implement at least part of a graphics rendering pipeline. For example, a TPC 320 can be configured to run a vertex shader program on the programmable streaming multiprocessor (SM) 340. The pipeline manager 310 can also be configured to forward packets received by the Xbar 270 to the appropriate logical units within the GPC 250.For example, some packets can be forwarded to fixed-function hardware units in the PROP 315 and / or the Raster Engine 325, while other packets can be forwarded to the TPCs 320 for processing by the Primitive Engine 335 or the SM 340.

[0067] The PROP unit 315 is configured to forward the data generated by the raster engine 325 and the TPCs 320 to a Raster Operations (ROP) unit in the memory partition unit 280, which is described in more detail below. The PROP unit 315 can also be configured to perform color mixing optimizations, organize pixel data, perform address translations, and the like.

[0068] The Raster Engine 325 comprises a set of fixed-function hardware units configured to perform various rasterization operations. In one embodiment, the Raster Engine 325 includes an adjustment engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile coalescing engine. The adjustment engine receives transformed vertices and generates plane equations associated with the geometric primitive defined by the vertices. The plane equations are then passed to the coarse raster engine to generate cover information (e.g., an x,y cover mask for a tile) for the primitive.The output of the coarse raster engine can be passed to the culling engine, where fragments associated with the primitive that fail a z-test are removed, and then to a clipping engine, where fragments outside a view volume are clipped. Fragments that survive clipping and culling can be passed to a fine raster engine to generate attributes for the pixel fragments based on the plane equations generated by the settings engine. The output of Raster Engine 380 includes fragments that, for example, are to be processed by a fragment shader implemented in a TPC 320.

[0069] Each TPC 320 contained in the GPC 250 includes an M-Pipe Controller (MPC) 330, a Primitive Engine 335, an SM 340, and one or more Structural Units 345. The MPC 330 controls the operation of the TPC 320 and forwards packets received by the Pipeline Manager 310 to the appropriate units of the TPC 320. For example, packets associated with a vertex can be forwarded to the Primitive Engine 335, which is configured to retrieve the vertex attributes associated with that vertex from memory 204. Conversely, packets associated with a shader program can be forwarded to the SM 340.

[0070] In one embodiment, the structure units 345 are configured to load structure maps (e.g., a 2D array of structures) from memory 204 and sample the structure maps to generate sampled structure values ​​for use in shader programs executed by the SM 340. The structure units 345 implement structure operations, such as filter operations, using mip maps (i.e., structure maps with varying levels of detail). In one embodiment, each TPC 320 includes two (2) structure units 345.

[0071] The SM 340 comprises a programmable streaming processor configured to handle tasks represented by a set of threads. Each SM 340 is multi-threaded and configured to execute a large number of threads (e.g., 32 threads) from a given group of threads concurrently. In one embodiment, the SM 340 implements a SIMD (Single Instruction, Multiple Data) architecture, where each thread in a group of threads (i.e., a warp) is configured to process a different data set based on the same instruction set. All threads in the group of threads execute the same instructions.In another embodiment, the SM 340 implements a SIMT (Single-Instruction, Multiple Thread) architecture, where each thread in a group of threads is configured to process a different data set based on the same instruction set, but individual threads within the group can deviate from this during execution. In other words, when an instruction is allocated to the group of threads for execution, some threads in the group may be active and thus execute the instruction, while other threads in the group may be inactive, resulting in a no-operation (NOP) instead of the instruction being executed. The SM 340 can be further described below in conjunction with... Fig. 4 will be described in more detail.

[0072] The MMU 390 provides an interface between the GPC 250 and the memory partition unit 280. The MMU 390 can provide virtual address to physical address translation, memory protection, and mediation of memory requests. In one embodiment, the MMU 390 provides one or more translation lookaside buffers (TLBs) to improve the translation of virtual addresses to physical addresses in memory 204.

[0073] Fig. 3B represents a storage partition unit 280 of the PPU 200. Fig. 2 according to one embodiment. As in Fig. As shown in Figure 3B, the memory partition unit 280 includes a raster operations (ROP) unit 350, a level 2 (L2) cache 360, a memory interface 370, and an L2 coupling array (XBar) 365. The memory interface 370 is coupled to the memory 204. The memory interface 370 can implement 16, 32, 64, 128-bit data buses, or the like, for high-speed data transmission. In one embodiment, the PPU 200U comprises memory interfaces 370, one memory interface 370 per memory partition unit 280, with each memory partition unit 280 being connected to a corresponding memory module 204. For example, the PPU 200 can be connected to up to U memory modules 204, such as a graphics card with double the data rate, version 5, synchronous dynamic random access memory (GDDR5 SDRAM). In one embodiment, the memory interface 370 implements a DRAM interface and U is equal to 6.

[0074] In one embodiment, the PPU 200 implements a multi-level memory hierarchy. The memory 204 is located off-chip in the SDRAM coupled to the PPU 200. Data from the memory 204 can be retrieved and stored in the L2 cache 360, which is located on-chip and shared by the various GPCs 250. As shown, each memory partition unit 280 contains a portion of the L2 cache 360, which is allocated to a corresponding memory element 204. Lower-level caches can then be implemented in different units within the GPCs 250. For example, each of the SMs 340 can implement a Level One (L1) cache. The L1 cache is private memory reserved for a specific SM 340. Data from the L2 cache 360 ​​can be retrieved and stored in each of the L1 caches for processing in the functional units of the SMs 340.The L2 cache 360 ​​is coupled with the memory interface 370 and the XBar 270.

[0075] The ROP unit 350 includes a ROP manager 355, a color ROP (CROP) unit 352, and a Z ROP (ZROP) unit 354. The CROP unit 352 performs raster operations related to pixel colors, such as color compression, pixel blending, and the like. The ZROP unit 354, in conjunction with the raster engine 325, performs depth checks. The ZROP unit 354 receives a depth for a pattern position assigned to a pixel fragment from the culling engine of the raster engine 325. The ZROP unit 354 tests this depth against a corresponding depth in a depth buffer for the pattern position assigned to the fragment. If the fragment passes the depth check for the pattern position, the ZROP unit 354 updates the depth buffer and sends a depth check result to the raster engine 325. The ROP manager 355 controls the operation of the ROP unit 350.It should be noted that the number of storage partition units 280 can differ from the number of GPCs 250, and therefore each ROP unit 350 can be coupled to any of the GPCs 250. Therefore, the ROP manager 355 tracks the packets received by the various GPCs 250 and determines to which GPC 250 a result generated by the ROP unit 350 is forwarded. The CROP unit 352 and the ZROP unit 354 are coupled to the L2 cache 360 ​​via an L2 XBar 365.

[0076] Fig. 4 introduces the 340 streaming multiprocessor from Fig. 3A according to one embodiment. As shown in Fig. As shown in Figure 4, the SM 340 includes an instruction cache 405, one or more flow control units 410, a register file 420, one or more processing cores 450, one or more special function units (SFUs) 452, one or more load / memory units (LSUs) 454, a link network 480 and a shared memory / L1 cache 470.

[0077] As described above, the Computing Workload Unit 225 and the Graphics Primitive Distribution Unit 230 allocate tasks for execution on the GPCs 250 of the PPU 200. The tasks are assigned to a specific TPC 320 within a GPC 250, and if the task is associated with a shader program, it may be assigned to an SM 340. A Flow Control Unit 410 receives the tasks from the Computing Workload Unit 225 and the Graphics Primitive Distribution Unit 230 and manages flow control for one or more groups of threads (i.e., warps) assigned to the SM 340. Each SM 340 can have K Flow Control Units 410 (i.e., 410(0) ... 410(K-1)). The Flow Control Unit 410 schedules threads for execution in groups of parallel threads, each group being called a warp. In one embodiment, each warp contains 32 threads.The 410 sequence control unit can manage a variety of different warps, schedule threads for execution, and then, during each clock cycle, allocate instructions from the variety of different warps to the various functional units (i.e., cores 350, SFUs 352, and LSUs 354).

[0078] In one embodiment, each sequence control unit 410 includes one or more command distribution units 415. Each distribution unit 415 is configured to transmit instructions to one or more of the functional units. In the Fig. In the embodiment shown in Figure 4, the sequence control unit 410 includes two distribution units 415, which enable two different instructions to be allocated from the same warp during each clock cycle. In alternative embodiments, each distribution unit 410 can have a single distribution unit 415 or additional distribution units 415.

[0079] Each SM 340 contains a register file 420, which provides a set of registers for the functional units of the SM 340. In one embodiment, the register file 420 is partitioned among the respective functional units, so that each functional unit is allocated a specific portion of the register file 420. In another embodiment, the register file 420 is partitioned among the various warps executed by the SM 340. The register file 420 provides temporary storage for operands associated with the data paths of the functional units.

[0080] Each SM 340 comprises L processing cores 450 (i.e., 450(0) ... 450(L-1)). In one embodiment, the SM 340 includes a large number (e.g., 192, etc.) of different processing cores 450. Each core 450 can include a single-precision, full-pipeline processing unit containing a floating-point arithmetic logic unit and an integer arithmetic logic unit. The core 450 can also include a double-precision processing unit with floating-point arithmetic. In one embodiment, the floating-point arithmetic logic units implement the IEEE 754-2008 standard for floating-point arithmetic. Each SM 340 also includes M SFUs 452 (i.e. 452(0) .... 452(M-1)), which perform special functions (e.g. pixel blending operations and the like), and N LSUs 454 (i.e. 454(0) ... 454(N-1)), which implement load and store operations between the common memory / L1 cache 470 and the register file 420.In one embodiment, the SM 340 includes 192 cores 450, 32 SFUs 452 and 32 LSUs 454.

[0081] Each SM 340 includes a connection network 480 that connects each of the functional units to the register file 420 and the shared memory / L1 cache 470. In one embodiment, the connection network 480 is a coupling array that can be configured to connect each of the functional units to one of the registers in the register file 420 or to the memory locations in the shared memory / L1 cache 470.

[0082] The shared memory / L1 cache 470 is an array of on-chip memory that, in one embodiment, can be configured either as shared memory, an L1 cache, or a combination of both, depending on the application requirements. For example, the shared memory / L1 cache 470 can comprise 64 kB of memory capacity. The shared memory / L1 cache 470 can be configured as either 64 kB of shared memory, 64 kB of L1 cache, or a combination of both, such as 16 kB of L1 cache and 48 kB of shared memory.

[0083] The PPU 200 described above can be configured to perform highly parallel calculations much faster than conventional CPUs. Parallel computing offers advantages in graphics processing, data compression, biometrics, stream processing algorithms, and similar applications.

[0084] In one embodiment, the PPU 200 includes a graphics processing unit (GPU). The PPU 200 is configured to receive instructions that specify shader programs for processing graphics data. Graphics data can be defined as a series of primitives, such as points, lines, triangles, quads, triangular strips, and the like. Typically, a primitive includes data specifying a number of vertices for the primitive (e.g., in a model-space coordinate system) as well as attributes associated with each vertex of the primitive. The PPU 200 can be configured to process the graphics primitives to generate a frame buffer (i.e., pixel data for each of the display's pixels).

[0085] An application writes model data for an image (i.e., a collection of vertices and attributes) to memory, such as system memory or memory 204. The model data defines each of the objects that can be visible on a display. The application then makes an API call to the driver kernel, which requests that the model data be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations for processing the model data. The commands can reference various shader programs to be implemented on the SMs 340 of the PPU 200, including one or more vertex shaders, hull shaders, domain shaders, geometry shaders, and pixel shaders. For example, one or more of the SMs 340 can be configured to run a vertex shader program that processes a number of vertices defined by the model data.In one embodiment, the various SMs 340 can be configured to run different shader programs concurrently. For example, a first subset of SMs 340 can be configured to run a vertex shader program, while a second subset of SMs 340 can be configured to run a pixel shader program. The first subset of SMs 340 processes vertex data to generate processed vertex data and writes the processed vertex data to the L2 cache 360 ​​and / or memory 204. After the processed vertex data has been rasterized (i.e., converted from three-dimensional data to two-dimensional data in screen space) to generate fragment data, the second subset of SMs 340 executes a pixel shader to generate processed fragment data, which is then blended with other processed fragment data and written to the frame buffer in memory 204.The vertex shader program and the pixel shader program can be executed simultaneously, processing different data from the same image in a pipeline until all model data for the image has been transferred to the frame buffer. The contents of the frame buffer are then transferred to a display controller for display on a screen.

[0086] The PPU 200 can be included in a desktop computer, laptop computer, tablet computer, smartphone (e.g., a wireless wearable device), personal digital assistant (PDA), digital camera, portable electronic device, and the like. In one embodiment, the PPU 200 is implemented on a single semiconductor substrate. In another embodiment, the PPU 200 is included in a system-on-a-chip (SoC) together with one or more other logic units, such as a reduced instruction set processor (RISC), a memory management unit (MMU), a digital-to-analog converter (DAC), and the like.

[0087] In one embodiment, the PPU 200 can be included on a graphics card that incorporates one or more memory modules 204, such as GDDR5 SDRAM. The graphics card can be configured to communicate with a PCIe slot on a desktop computer motherboard, which may include, for example, a northbridge and a southbridge chipset. In yet another embodiment, the PPU 200 can be an integrated graphics processing unit (iGPU) included in the motherboard chipset (e.g., the northbridge).

[0088] Fig. Figure 5 presents an exemplary System 500 in which the different architectures and / or functionalities of the various preceding embodiments can be implemented. As shown, a System 500 is provided that includes at least one central processor 501 connected to a communication bus 502. The communication bus 502 can be implemented with any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol(s). The System 500 also includes a main memory 504. The control logic (software) and the data are stored in the main memory 504, which can be implemented as random-access memory (RAM).

[0089] The System 500 also includes input devices 512, a graphics processor 506, and a display 508, i.e., a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light-emitting diode), plasma display, or the like. User input can be received by the input devices 512, e.g., keyboard, mouse, touchpad, microphone, and the like. In one embodiment, the graphics processor 506 can include a variety of shader modules, a raster module, etc. Each of the aforementioned modules can even be located on a single semiconductor platform to form a graphics processing unit (GPU).

[0090] In this description, a single semiconductor platform can refer to a single, unified semiconductor-based integrated circuit or chip. It should be noted that the term "single semiconductor platform" can also refer to multi-chip modules with enhanced connectivity that simulate on-chip operation and offer significant improvements over the use of a conventional central processing unit (CPU) and bus implementation. Naturally, the various modules can also be arranged individually or in various combinations of semiconductor platforms, according to the user's requirements.

[0091] The System 500 can also include a Secondary Storage 510. The Secondary Storage 510 includes, for example, a hard disk drive and / or a removable storage drive, which can be a floppy disk drive, a magnetic tape drive, a compact floppy disk drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a standard manner.

[0092] Computer programs or algorithms of computer control logic can be stored in main memory 504 and / or secondary memory 510. Such computer programs, when executed, enable the System 500 to perform various functions. Memory 504, memory 510, and / or any other memory are possible examples of media readable by a computer.

[0093] In one embodiment, the architecture and / or functionality of the various preceding figures may be implemented within the central processing unit 501, the graphics processing unit 506, an integrated circuit (not shown) that can fulfill at least some of the capabilities of both the central processing unit 501 and the graphics processing unit 506, a chipset (i.e., a group of integrated circuits that are intended to function and be sold as a unit for performing related functions, etc.), and / or another integrated circuit.

[0094] Furthermore, the architecture and / or functionality of the various preceding figures can be implemented within the context of a general-purpose computer system, a circuit board system, a game console system for entertainment purposes, an application-specific system, and / or any other desired system. For example, System 500 can take the form of a desktop computer, laptop, server, workstation, game console, embedded system, and / or other type of logic. However, System 500 can also take the form of various other devices, including, but not limited to, a PDA (Personal Digital Assistant), a mobile phone device, a television, and so on.

[0095] Furthermore, although not shown, the System 500 can be coupled to a network (e.g., telecommunications network, Local Area Network (LAN), Wireless Network, Wide Area Network (WAN) such as the Internet, Peer-to-Peer network, cable network, or the like) for communication purposes.

[0096] Fig. Figure 6 is a conceptual representation of a graphics processing pipeline 600, which originates from the PPU 200. Fig. 2 was implemented according to one embodiment. The graphics processing pipeline 600 is an abstract flowchart of the implemented processing steps for generating 2D computer images from 3D geometry data. As is known, pipeline architectures can perform long-latency operations more efficiently by dividing the operation into a plurality of stages, with the output of each stage coupled to the input of the next subsequent stage. Thus, the graphics processing pipeline 600 receives input data 601, which is transferred from one stage to the next stage of the graphics processing pipeline 600 to generate output data 602. In one embodiment, the graphics processing pipeline 600 can represent a graphics processing pipeline defined by the O-penGL@ API.Optionally, the graphics processing pipeline 600 can be implemented within the functionality and architecture of the previous figures and / or any subsequent figure(s).

[0097] As in Fig. As shown in Figure 6, the graphics processing pipeline 600 comprises a pipeline architecture that includes several stages. The stages include, but are not limited to, a data assembly stage 610, a vertex shading stage 620, a primitive assembly stage 630, a geometry shading stage 640, a viewport, scale, cull, and clip (VSCC) stage 650, a raster stage 660, a fragment shading stage 670, and a raster operations stage 680. In one embodiment, the input data 601 includes instructions that configure the processing units to implement the stages of the graphics processing pipeline 600 and geometric primitives (e.g., points, lines, triangles, quadrilaterals, quads, triangular strips, or fans, etc.) to be processed by the stages. The output data 602 can include pixel data (i.e., color data) that is copied into a frame buffer or some other type of surface data structure in memory.

[0098] The data assembly stage 610 receives the input data 601, which specifies vertices for high-order surfaces, primitives, or the like. The data assembly stage 610 collects the vertex data in a buffer or queue, for example, by receiving a command from the host processor containing a pointer to a buffer in memory, and reads the vertex data from the buffer. The vertex data is then transferred to the vertex shading stage 620 for processing.

[0099] The vertex shading stage 620 processes vertex data by performing a series of operations (e.g., a vertex shader or a program) once for each of the vertices. Vertices can be, for example, a vector with 4 coordinates (i.e.,<x, y, z, w> The vertex shading level 620 can manipulate individual vertex attributes such as position, color, structural coordinates, and the like. In other words, the vertex shading level 620 performs operations on the vertex coordinates or other vertex attributes associated with a vertex. Such operations typically include lighting operations (i.e., changing color attributes for a vertex) and transformation operations (i.e., changing the coordinate space for a vertex).For example, vertices can be specified by coordinates in an object coordinate space, which are transformed by multiplying the coordinates by a matrix that translates the coordinates from the object coordinate space into a worldspace or a normalized device coordinate space (NDC). The vertex shading stage 620 produces transformed vertex data, which is then transferred to the primitive assembly stage 630.

[0100] The primitive assembly stage 630 collects the vertices output by the vertex shading stage 620 and groups them into geometric primitives for processing by the geometry shading stage 640. For example, the primitive assembly stage 630 can be configured to group three consecutive vertices at a time into a geometric primitive (i.e., a triangle) for transfer to the geometry shading stage 640. In some embodiments, certain vertices can be reused for subsequent geometric primitives (e.g., two consecutive triangles in a triangle stripe can share two vertices). The primitive assembly stage 630 transfers geometric primitives (i.e., a collection of associated vertices) to the geometry shading stage 640.

[0101] The geometry shading stage 640 processes geometric primitives by performing a series of operations (i.e., a geometry shader or program) on them. Tessellation operations can create one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 640 can subdivide each geometric primitive into a finer mesh of two or more geometric primitives for processing by the rest of the graphics processing pipeline 600. The geometry shading stage 640 then passes geometric primitives to the viewport SCC stage 650.

[0102] In one embodiment, the graphics processing pipeline 600 can be operated within a streaming multiprocessor, and the vertex shading stage 620, the primitive assembly stage 630, the geometry shading stage 640, the fragment shading stage 670, and / or the associated hardware and software can perform processing operations sequentially. Once the sequential processing operations are complete in one embodiment, the viewport SCC stage 650 can use the data. In one embodiment, primitive data processed by one or more of the steps in the graphics processing pipeline 600 can be written to a cache (e.g., an L1 cache, vertex cache, etc.). In this case, the viewport SCC stage 650 can access the data in the cache in one embodiment. In one embodiment, the viewport SCC stage 650 and the raster stage 660 are designed as a circuit with a defined function.

[0103] The Viewport SCC level 650 performs scaling, culling, and clipping of geometric primitives. Each surface to be rendered onto is associated with an abstract camera position. The camera position represents the position of a viewer looking at the image and defines a view volume that includes the objects in the image. The view volume can have a viewing plane, a back plane, and four clipping planes. Any geometric primitive that lies entirely outside the view volume can be culled (i.e., discarded) because it does not contribute to the final rendered image. Any geometric primitive that lies partly inside and partly outside the view volume can be culled (i.e., converted into a new geometric primitive that is enclosed within the view volume).Furthermore, geometric primitives can be scaled depending on the depth of the viewing volume. All potentially visible geometric primitives are then transferred to the 660 pixel grid level.

[0104] Raster stage 660 converts the 3D geometric primitives into 2D fragments (e.g., suitable for use in rendering, etc.). Raster stage 660 can be configured to use the vertices of the geometric primitives to establish a set of plane equations from which various attributes can be interpolated. Raster stage 660 can also compute a cover mask for a multitude of pixels, indicating whether one or more sample locations for the pixel interrupt the geometric primitive. In one embodiment, a Z-test can also be performed to determine whether the geometric primitive is occluded by other geometric primitives that have already been rasterized. Raster stage 660 produces fragment data (i.e., interpolated vertex attributes associated with a specific sample location for each covered pixel), which is passed to fragment shading stage 670.

[0105] Fragment shading stage 670 processes fragment data by performing a series of operations (i.e., a fragment shader or program) on each fragment. Fragment shading stage 670 can generate pixel data (i.e., color values) for the fragment, for example, through lighting operations or sampling structure maps using interpolated structure coordinates for the fragment. Fragment shading stage 670 generates pixel data that is then passed to raster operations stage 680.

[0106] Raster operations stage 680 can perform various operations on the pixel data, such as alpha testing, stencil testing, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. Once raster operations stage 680 has completed processing the pixel data (i.e., the output data 602), the pixel data can be written to a render target, such as a frame buffer, color buffer, or similar.

[0107] It should be noted that in addition to or instead of one or more of the stages described above, one or more additional stages may be included in the graphics processing pipeline 600. Different implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, in some embodiments, one or more of the stages described above may be excluded from the graphics processing pipeline (e.g., the geometry shading stage 640). Other types of graphics processing pipelines are considered to be within the scope of this disclosure. Moreover, each of the stages of the graphics processing pipeline 600 may be implemented by one or more dedicated hardware units within a graphics processing unit such as the PPU 200. Additional stages of the graphics processing pipeline 600 may be implemented by programmable hardware units such as the SM 340 of the PPU 200.

[0108] The 600 graphics processing pipeline can be implemented by an application running on a host processor, such as a CPU 501. In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that an application can use to generate graphical data for a display. The device driver is a software program containing a variety of instructions that control the operation of the 200 PPU. The API provides an abstraction for a programmer, allowing them to use specialized graphics hardware, such as the 200 PPU, to generate the graphical data without having to use the specific instruction set for the 200 PPU. The application can include an API call that is forwarded to the device driver for the 200 PPU.The device driver interprets the API call and performs various operations to respond to it. In some cases, the device driver can perform operations by executing instructions on the CPU 501. In other cases, the device driver can perform at least some operations by initiating operations on the PPU 200 via an input / output interface between the CPU 501 and the PPU 200. In one embodiment, the device driver is designed to implement the graphics processing pipeline 600 using the hardware of the PPU 200.

[0109] Within the PPU 200, various programs can be executed to implement the different stages of the 600 graphics processing pipeline. For example, the device driver can start a kernel on the PPU 200 to execute the vertex shading stage 620 on an SM 340 (or multiple SMs 340). The device driver (or the original kernel running on the PPU 200) can also start other kernels on the PPU 200 to execute other stages of the 600 graphics processing pipeline, such as the geometry shading stage 640 and the fragment shading stage 670. Furthermore, some stages of the 600 graphics processing pipeline can be implemented on dedicated hardware, such as a rasterizer or a data assembler implemented within the PPU 200.It should be noted that the results of one core may be processed by one or more intermediate hardware units with a defined function before being processed by a subsequent core on an SM 340.

[0110] Fig. Figure 7A is a conceptual representation of a two-processor configuration 700 according to one embodiment. As shown, a first GPU 139(0) and a second GPU 139(1) are configured to communicate via a high-speed connection 150 and a system interface such as a PCIe interface 710. Communication via the high-speed connection 150 includes, without limitation, the transfer, copying, and / or moving of data pages from a source memory circuit on one GPU 139 to a destination memory circuit on the other GPU 139. The data may be represented in a compressed or uncompressed format. In one embodiment, the PCIe interface 710 is coupled to a CPU (not shown), and the CPU is configured to execute at least one driver for the GPUs 139.The at least one driver is designed to execute various procedures described herein, such as procedure 120 of . Fig. 1C. Furthermore, the GPUs are designed to support the 100 process. Fig. 1A and / or procedure 110 of Fig. 1B to be executed.

[0111] In one embodiment, the first GPU and the second GPU are coupled to each other via a first high-speed connection 150 (e.g., NVLINK) and configured to render frames for a first display device (not shown). In another embodiment, which is described in Fig. As shown in Figure 7B, a third and a fourth GPU are coupled together via a second high-speed connection 150(1) and configured to display frames for a second display device. The first through fourth GPUs can be further coupled together by using another type of high-speed connection, shown here as a PCI Express (PCIe) interface 710, to provide communication between the GPUs.

[0112] Fig. Figure 7B is a conceptual representation of a four-processor configuration 702 according to one embodiment. As shown, a first GPU 139(0) and a second GPU 139(1) are configured to communicate via a first high-speed link 150(0) and a system interface such as a PCIe interface 710. Furthermore, a third GPU 139(2) and a fourth GPU 139(3) are configured to communicate via a second high-speed link 150(1) and a system interface such as the PCIe interface 710. Communication via the high-speed links 150(0), 150(1) includes, without limitation, the transfer, copying, and / or moving of data pages from a source memory circuit on one GPU 139 to a destination memory circuit on another GPU 139.In one embodiment, the PCIe interface 710 is coupled to a CPU (not shown), and the CPU is configured to execute at least one driver for the GPUs 139. The at least one driver is configured to execute various methods described herein, such as method 120 of . Fig. 1C. Furthermore, the GPUs are designed to support the 100 process. Fig. 1A and / or procedure 110 of Fig. 1B. While two GPUs 139 are shown coupled to each high-speed link 150, different embodiments can implement different high-speed link configurations that accommodate more than two GPUs 139.

[0113] Fig. Figure 7C is a conceptual representation of a second four-processor configuration 704 according to one embodiment. As shown, four GPUs 139(0), 139(1), 139(2), and 139(3) are configured to communicate via a high-speed link 150(0) and a system interface such as a PCIe interface 710. Communication via the high-speed link 150 includes, without limitation, the transfer, copying, and / or moving of data pages from a source memory circuit on one GPU 139 to a destination memory circuit on another GPU 139. In one embodiment, the PCIe interface 710 is coupled to a CPU (not shown), and the CPU is configured to execute at least one driver for the GPUs 139. The at least one driver is configured to execute various methods described herein, such as method 120 of Fig. 1C. Furthermore, the GPUs are designed to support the 100 process. Fig. 1A and / or procedure 110 of Fig. 1B. While four GPUs 139 are shown coupled to the high-speed link 150, different embodiments can implement different high-speed link configurations that accommodate three GPUs 139 or more than four GPUs 139.

[0114] In one embodiment, GPUs 139(0) and GPU 139(1) are configured to render a first frame. In another embodiment, GPUs 139(2) and GPU 139(3) are configured to render a second frame. In certain embodiments, the first frame is displayed on the left side of a virtual reality display system, and the second frame is displayed on the right side of the virtual reality display system. In certain embodiments, AFR (Automatic Frame Layout) is implemented with GPUs 139(0) to 139(3). For example, GPUs 139(0) and 139(1) can cooperate using checkerboard work allocation to render even-numbered frames, and GPUs 139(2) and 139(3) can cooperate using checkerboard work allocation to render odd-numbered frames.In another example, GPUs 139(0) and 139(1) can work together using the checkerboard work assignment to render frames for the left eye in a virtual reality system, and GPUs 139(2) and 139(3) can work together using the checkerboard work assignment to render frames for the right eye in the virtual reality system.

[0115] Distributing rendering across multiple GPUs reduces rendering latency and provides a general-purpose and scalable GPU computing environment compared to traditional techniques. Access to data stored in remote memory is achieved via a high-speed data link coupled to the multiple GPUs. Hardware access counters in each GPU are designed to identify frequently accessed data pages retrieved from remote memory while a frame is being rendered, triggering a notification to an associated driver. In response to this notification, the driver schedules an operation to copy the identified data page from remote memory to the GPU's local memory before rendering a subsequent frame.The hardware access counter can selectively count access requests to the same data page, based on the rendering pass, the context, and additional conditions.

[0116] In one embodiment, a local L2 cache is designed to consolidate requests before a write-back operation is performed, thereby improving transfer efficiency. Similarly, the local L2 cache can combine adjacent requests, also to improve transfer efficiency, by combining several small requests into a larger block transfer. Specifically, this L2 cache behavior can be implemented by configuring the L2 cache to use a write-back policy for a peer access or peer aperture.

[0117] In certain implementations, the driver is designed to occasionally experiment by adjusting settings and recording which settings are more advantageous for future use. For example, the driver might try using a surface in a given run via direct remote access and then via a previously created copy. By measuring the performance for both surface access configurations, the driver can select the faster configuration for normal or extended operation. Another experiment the driver can perform is to allocate all rendering to all GPUs, as opposed to distributing the workload. While this allocation clearly doesn't allow for scaling, it also has no inherent drawbacks and is advantageous whenever a run requires little processing power and produces large amounts of data. Shadow map rendering is one such example.When the work is distributed, the complete shadow map is available on both GPUs for later use.

[0118] The driver can decide that certain passes should be executed entirely on one GPU and other passes entirely on another. Shadow map rendering is an example of how this approach can be advantageous. In a scenario where there are N light sources in an image, each requiring N shadow maps, one GPU can generate N / 2 shadow maps, while another GPU generates the remaining shadow maps. While the GPUs are rendering the next set of shadow maps, the completed shadow maps can be asynchronously copied to the other GPUs (e.g., using the Copy Engine (CE)).

Claims

[1] Procedure encompassing: Generating, by a first graphics processing unit, GPU (139(0)) coupled to a first memory circuit, one or more first memory access requests in connection with rendering a first primitive for a first frame, wherein at least one of the first memory access requests targets a first data page which is physically located in a second memory circuit coupled to a second GPU (139(1)); Request, by the first GPU(139(0)), the first data page via a first data link that couples the first GPU to the second GPU; Collecting, through a register circuit in the first GPU (139(0)), an access request count for the first data page; Notification of a driver by the first GPU (139(0)) that the access request count has reached a certain threshold, Received, by the first GPU (139(0)), a first copy instruction to copy the first data page from the second memory circuit via the first data link, to create a copy of the first data page in the first memory circuit before the first GPU (139(0)) accesses the first data page in connection with rendering the first primitive for a second frame, Execution of the first copy command by the first GPU (139(0)); and Generating, by the first GPU (139(0)), one or more second memory access requests in connection with rendering the first primitive, wherein at least one of the second memory access requests targets the copy of the first data page in the first memory circuit. [2] Method according to claim 1, wherein the first data page is stored in a compressed format in the second storage circuit and the copy of the first data page is stored in the compressed format in the first storage circuit. [3] Method according to claim 2, wherein the first data page is copied through the first data connection in the compressed format. [4] Method according to any of the preceding claims, wherein a first command stream specifies a first render pass for the first frame, and a second command stream specifies a first render pass for a second frame, and wherein the notification occurs during the first frame. [5] Method according to any of the preceding claims, further comprising, prior to generating one or more first memory access requests: Receiving the first primitive by the first GPU (139(0)); and Determine by a clipping circuit in the first GPU (139(0)) that a layer for the first primitive cuts a first area of ​​the first frame which is assigned to the first GPU. [6] Method according to any of the preceding claims, wherein requesting the first data page through the first data connection comprises determining that the first data page is located in a first remote access which is assigned to the second GPU (139(1)). [7] Method according to claim 6, wherein a memory management unit determines that the first page is located in the first detached access. [8] Method according to any of the preceding claims, wherein the one or more first memory access requests comprise an atomic access operation performed on data located in the second memory circuit. [9] Method according to any of the preceding claims, further comprising, prior to generating one or more first memory access requests: Receiving the first primitive by the first GPU (139(0)); and Determine, through preceding shader instructions, that a first cooperative thread arrangement, which includes the first primitive, is executed on the first GPU. [10] Method according to any of the preceding claims, wherein the one or more first memory access requests each have a memory address; and wherein the register circuit is configured to increment the number of access requests when the memory address is in a programmable address range for the register circuit. [11] Method according to any of the preceding claims, wherein the first frame is divided into rectangular areas and adjacent rectangular areas which have a common edge are alternately assigned to the first GPU (139(0)) and the second GPU (139(1)). [12] Method according to claim 11, wherein the rectangular areas assigned to the first GPU form a checkerboard pattern. [13] Processing system which is designed, to generate one or more first memory access requests in connection with rendering a first primitive for a first frame by means of a first graphics processing unit, GPU, (139(0)) coupled to a first memory circuit, wherein at least one of the first memory access requests targets a first data page which is physically located in a second memory circuit coupled to a second GPU (139(1)); to request the first data page via a first data connection that couples the first GPU with the second GPU using the first GPU; to collect a count of access requests for the first data page using a register circuit in the first GPU; to notify a driver via the first GPU that the number of access requests has reached a certain threshold, to receive a first copy command through the first GPU, to copy the first data page from the second memory circuit via the first data link, to create a copy of the first data page in the first memory circuit before the first GPU accesses the first data page in connection with rendering the first primitive for a second frame, to execute the first copy command using the first GPU; and by the first GPU to generate one or more second memory access requests in connection with rendering the first primitive, wherein at least one of the second memory access requests targets the copy of the first data page in the first memory circuit. [14] Processing system according to claim 13, which is further configured to carry out a method according to any one of claims 2 to 12. [15] Processing system according to claim 13 or 14, further comprising a cache auxiliary system configured to combine two or more of the first memory access requests into one request. [16] Non-volatile, computer-readable storage medium which stores instructions which, when executed by a processor, cause the processor to execute a method according to any one of claims 1 to 12.

Images