Shared data fabric processing client reset system and method

Abstract

A processing system including a shared data fabric resets a first client processor while operating a second client processor. The first client processor is instructed to stop requests to one or more devices of the shared data fabric. Status communication is blocked between the first client processor and a memory controller, a second client processor, or both, such that the first client processor enters a temporary offline state. The first client processor is designated as non-coherent. Thus, when the first client processor is reset, certain errors and efficiency losses due to messages sent during or before the reset are prevented.

Description

[Background technology]

[0001] A shared data fabric system may include a variety of heterogeneous processing clients, including various central processing units (CPUs), graphic processing units (GPUs), digital signal processors (DSPs), other parallel processors, other special-purpose devices, or any combination thereof. Some processing clients, if standalone, such as GPUs, can be reset during system operation without causing errors in other components because these components do not share the same coherent data fabric as other processing clients. However, in a shared data fabric system, processing clients frequently send status communications, including probes and status requests, to each other, and resetting a processing client while a status communication is outstanding may cause unintended consequences, such as system failure or degraded system performance.

[0002] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference numbers in different drawings indicates similar or identical items. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram of a shared data fabric processing system including multiple processing clients, according to some embodiments. [Figure 2] FIG. 1 is a flow diagram of a method for resetting a processing client in a shared data fabric processing system, according to some embodiments. [Figure 3] FIG. 1 is a block diagram of a portion of a shared data fabric processing system that uses processing credits to send packets to processing clients, according to some embodiments. [Figure 4] FIG. 1 is a block diagram of a portion of a shared data fabric processing system that manages probes, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0004] The processing system includes a shared data fabric connecting multiple processing clients. The processing clients send status communications (e.g., packets and probes) to each other and to various other devices in the processing system, such as a memory controller. In some embodiments, as described herein, the processing system resets a first one of the processing clients (e.g., a parallel processing unit such as a GPU) without resetting the entire system or other processing clients (e.g., CPUs). In particular, the processing system intercepts, blocks, and / or spoofs status communications between the first processing client and other devices in the processing system. As a result, the first processing client enters a temporary offline state, where the first processing client continues to service requests but the processing system sends status communications as if the first processing client were offline. Eventually, the first processing client enters a state where various other devices are not performing operations on behalf of the first processing client and the first processing client is not expected to send messages to various other devices. At that point, the first processing client is reset, and the processing system stops intercepting, blocking, and / or spoofing communications. The procedures described herein avoid various potential errors, losses of efficiency, or both, described below, despite the resetting of the first processing client.

[0005] In some cases, various problems arise as a result of a naive reset of a first processing client (e.g., a GPU). For example, if a CPU sends a probe request to a GPU and the GPU is reset before responding to the probe, in some cases, a response to the probe is not sent. As a result, in some cases, the CPU will experience a potentially fatal error if a response to the probe request is not received. As another example, if processing credits are allocated to the GPU before the GPU is reset and the GPU does not release those processing credits, in some cases, the GPU will not recognize those processing credits after the reset, preventing the processing credits from being used by the entire system and degrading system performance. As yet another example, if the GPU sends a communication (e.g., a status request) to the CPU before the GPU is reset, but the CPU does not respond to the communication until after the GPU is reset, after the reset, the GPU is no longer looking to receive a response. In some cases, the GPU will not recognize to send a communication. As a result, if the GPU receives a response to a communication from the CPU after the GPU has been reset, the GPU may register the communication as an error or may incorrectly register the response as a response to another communication.

[0006] The techniques described herein, in various embodiments, are utilized in any of a variety of parallel processors (e.g., vector processors, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), non-scalar processors, highly parallel processors, artificial intelligence (AI) processors, inference engines, machine learning processors, other multi-threaded processing units, etc.). For ease of explanation, reference will be made herein to an exemplary context in which a GPU is used. However, it will be understood that the systems and techniques described herein apply equally to the use of other types of parallel processors, unless otherwise noted.

[0007] 1 illustrates a processing system 100 for resetting a processing client, according to at least some embodiments. Processing system 100 includes a data fabric 102 that is used to interconnect various components of processing system 100, including multiple processing clients, such as processing clients 104-106, one or more memory controllers 108, and one or more I / O hubs 110. Each memory controller 108 is itself coupled to system memory 112, and each I / O hub 110 is coupled to one or more I / O devices, such as I / O devices 114 and 115.

[0008] Each of the processing clients 104-106 includes one or more processor cores and a local cache hierarchy. The processor core may include, for example, a CPU core, a GPU core, a DSP core, a parallel processor core, or a combination thereof. The local cache hierarchy of a processing client includes one or more levels of cache. In some embodiments, at least one of the processing clients 104-106 is different from at least one other of the processing clients 104-106 (i.e., the processing clients 104-106 are heterogeneous). In other embodiments, the processing clients 104-106 are homogeneous.

[0009] The memory controller 108 acts as an interface between the corresponding system memory 112 and other components of the processing system 100. As such, data cached in a processing client's local cache hierarchy is typically manipulated as blocks of data called "cache lines," which are addressed or otherwise located in the memory hierarchy using physical addresses in the system memory 112. Cache lines are accessed from the system memory 112 by the memory controller 108 in response to access requests from the processing clients, and the cache lines are installed or cached in one or more caches of the processing clients. Similarly, when a cache line containing modified data needs to be evicted from a processing client's local cache hierarchy and therefore updated in the system memory 112, the memory controller 108 manages this write-back process.

[0010] I / O devices 114, 115 operate to transfer data into and out of node 101 using DMA access operations. For example, any of I / O devices 114, 115 may include a network interface card (NIC) for connecting the node to a network to receive and transmit data, or a hard disk drive (HDD) or other mass storage device for non-volatile storage of relatively large amounts of data used by processing clients 104-106. In at least one embodiment, I / O hub 110 manages I / O devices 114, 115 and serves as an interface between data fabric 102 and I / O devices 114, 115. To illustrate, in some embodiments, I / O hub 110 includes a Peripheral Component Interconnect Express (PCIe) root complex to operate as a PCIe interconnect between I / O devices 114, 115 and data fabric 102.

[0011] The data fabric 102 includes a transport layer 130 and multiple coherent agents 134-136 that act as an interface between the transport layer 130 and each of the processing clients 104-106. Additionally, the memory controller 108 acts as a coherent agent for the system memory 112. However, in other embodiments, the memory controller 108 is located external to the data fabric 102, and separate coherent agents are included for the memory controller 108 and the system memory 112. The I / O agent 140 acts as a coherent agent for the I / O hub 110 and the I / O devices 114, 115. Furthermore, in some embodiments, the transport layer 130 is coupled to corresponding transport layers of one or more other nodes 141 via bridge components (not shown). In various embodiments, the data fabric 102 is compatible with one or more standardized interconnect specifications, such as the HyperTransport™ specification or the Infinity Fabric™ specification.

[0012] As a general operational overview, the data fabric 102 transfers commands, data, requests, status communications, and other signaling between other components of the processing system 100 and between the processing system 100 and other nodes 141. One such subset of these transport operations is the storage of data provided by the I / O devices 114, 115 for use by one or more of the processing clients 104-106. As described further below with reference to FIG. 3, another subset of these transport operations is the sending of packets (e.g., read requests and responses or write requests and responses) from the memory controller 108 or a processing client (e.g., processing client 104) to another processing client (e.g., processing client 106). As described further below with reference to FIG. 4, another subset includes status communications, such as probes, sent from the memory controller 108 or a processing client (e.g., processing client 104) to another processing client (e.g., processing client 106).

[0013] In some cases, due to an error, failure, or the like, the processing system 100 determines that a processing client should be reset. In various embodiments, this determination is made by the corresponding coherent agent, the memory controller 108, the processing client being reset, by another processing client, or by another device. As discussed above, in some cases, a naive reset of a processing client can cause various errors, failures, or loss of efficiency. Therefore, as further described below with reference to FIG. 2 , some or all of various steps can be taken to prevent problems associated with resetting a processing client. In some embodiments, the processing system 100 can reset any of the processing clients 104-106 without resetting the rest of the processing system 100. In other embodiments, one or more of the processing clients 104-106 cannot be reset without resetting the rest of the processing system 100.

[0014] 2 is a flow diagram illustrating a method 200 for resetting processing clients in a shared data fabric processing system, according to some embodiments. Method 200 is performed in some embodiments by a corresponding coherent agent, a memory controller, by some of the processing clients being reset, or by another processing client. In some embodiments, method 200 is initiated by one or more processors in response to one or more instructions stored by a computer-readable storage medium. In some cases, method 200 includes fewer blocks (e.g., flow control checks are not disabled because flow control checks are already disabled for another reason) or additional blocks.

[0015] In block 202, the processing clients are instructed to stop making requests to the data fabric. For example, processing client 106 stops making requests to memory controller 108 and processing client 104 of data fabric 102. In some cases, stopping making requests includes one or more of terminating chained writes and terminating requested writes. In various embodiments, this stopping of requests coincides with a clean request boundary.

[0016] In block 204, the processing client is instructed to enter a temporary offline state in which various communications, such as status communications and read and write requests, between the processing client and the memory controller, or between the processing client and other processing clients or devices in the processing system, are at least partially blocked or intercepted. For example, processing client 106 enters the temporary offline state. As a result of the processing client being in the temporary offline state, in some embodiments, the processing client appears to be offline to other processing clients, although incoming coherence messages from other processing clients continue to be serviced. In the illustrated embodiment, block 204 includes blocks 206-210.

[0017] In block 206, the processing credits of the processing client are wiped. For example, the processing credits tracked by the coherent agent 136 and used to prevent the processing client 106 from receiving more tasks than the processing client can execute in a particular time are treated as allocated, potentially preventing other devices in the processing system 100 from sending tasks to the processing client 106.

[0018] At block 208, flow control checks for the transaction client are disabled, for example, flow control checks for responses and probe requests to the transaction client 106 are disabled.

[0019] At block 210, spoofing of responses to probes sent to processing clients is enabled. For example, a spoofed probe response is generated by coherent agent 136 and sent in response to a probe sent from processing client 104 to processing client 106. As a result, the response to the probe sent to processing client 106 includes an indication of a cache miss.

[0020] At block 212, the processing system waits for a request by a processing client to flush out from the system. For example, the processing system 100 waits for read requests, write requests, status communications, probes, and other communications by the processing client 106 to be resolved. In various embodiments, the processing system waits 10 nanoseconds or 100 microseconds. In some embodiments, the amount of time to wait is stored in memory, such as the system memory 112 or the memory of the coherent agent 136.

[0021] At block 214, the processing client is marked as non-coherent. For example, processing client 106 is marked as non-coherent because its cache has been flushed. As a result, in the illustrated embodiment, probes no longer target processing client 106 because processing client 106 is no longer a cache hit. In some embodiments, method 400 waits for probes targeted at processing client 106 to complete (e.g., by sending a spoofed probe response).

[0022] In block 216, the processing client is reset. For example, in some cases, the processing client 106 is reset, which returns the interface of the processing client 106 to a reset initialization state. The coherent agent 136 is instructed to treat the processing client 106 as having a software fuse-off state. Thus, the processing client 106 resets even though other parts of the processing system 100 remain online. In some embodiments, exiting the fuse-off state as part of resetting the processing client resets the number of processing credits for the processing client.

[0023] In block 218, the processing client exits the temporary offline state. For example, various communications to the processing client 106 are no longer blocked or intercepted. In the illustrated embodiment, block 218 includes blocks 220, 222, and 223.

[0024] At block 220, spoofing of responses to probes sent to the processing client is disabled. For example, spoofed responses are no longer sent in response to probes sent to the processing client 106. At block 222, flow control checks are enabled. At block 223, the processing client's processing credits are re-released and available for allocation to other components of the processing system.

[0025] The processing client is marked as coherent in block 224. As a result, in some cases, probes in the processing system 100 will again target the processing client 106. This will reset the processing client.

[0026] 3 is a block diagram illustrating the use of processing credits for processing packets in the processing client 106 of FIG. 1 , according to some embodiments. In the illustrated embodiment, the coherent agent 136 includes a credit counter 302, a credit validation module 304, a selection module 306, and a packet memory 308. The processing client 106 includes a credit selection module 310, a packet selection module 312, and a packet memory 314. While the illustrated embodiment shows a particular configuration of components, other combinations of components are contemplated in various embodiments. Additionally, in some embodiments, additional components, such as buffers or repeaters, are contemplated but are not shown for ease of illustration.

[0027] The packet selection module 312 instructs the packet memory 314 to send the packet 346 to an addressed portion of the processing client 106 (e.g., a processor core, a memory device, etc.). In some embodiments, the packet selection module 312 specifies the packet to be sent. In other embodiments, the packet memory 314 is a queue and the next packet to be sent.

[0028] As described above, processing credits are used to manage requests allocated to a processing client 106. Accordingly, the packet selection module 312 sends a credit release signal 340 to the credit selection module 310 to indicate the number of credits to be released. The credit selection module 310 sends an indication of the available credits as a credit availability signal 336 to the credit counter 302. As described above with reference to FIG. 2, in some embodiments, resetting a processing client 106 includes releasing processing credits after the processing client 106 resets. An initial credit count 338 indicates the initial number of processing credits available before those processing credits are used. The credit selection module 310 selects to send the credit release signal 340 and the initial credit count 338 as a credit availability signal 336 to the credit counter 302 based on a credit reset signal 320 that is asserted as part of the reset process of the processing client 106, whereby the processing credits are re-released.

[0029] The credit counter 302 tracks the number of available processing credits for the processing client 106. In the illustrated embodiment, the number of available processing credits is increased in response to a credit availability signal 336 from the credit selection module 310, which indicates the number of newly available credits at the processing client 106. The number of available processing credits is decreased in response to a credit usage signal 324 from the selection module 306. As described above, as part of resetting a processing client 106, the processing credits for the processing client 106 are wiped. In some embodiments, a credit reset signal 320 indicates that no processing credits are available. In the illustrated embodiment, the credit reset signal 320 is sent to both the credit counter 302 and the credit selection module 310. In other embodiments, separate signals are sent to the credit counter 302 and the credit selection module 310. The credit counter 302 indicates whether credits are available to the credit activation module 304 via a credit indication 322.

[0030] The credit enablement module 304 selectively enables or disables processing of packets based on whether processing credits are available and whether the flow control signal 328 is enabled. As described above with reference to FIG. 2, in some embodiments, if credits are not available or the flow control signal 328 is disabled, which occurs as part of a processing client reset process, the credit enablement module 304 indicates via the selection permission signal 326 that the selection module 306 is authorized to select a communication (e.g., a packet or a probe) to send to the processing client 106.

[0031] In response to the selection grant signal 326, the selection module 306 indicates via a packet selection signal 332 that the packet memory 308 is authorized to send a packet to the processing client 106. In some embodiments, the selection module 306 specifies the packet to be sent. In other embodiments, the packet memory 308 is a queue, and the next packet is to be sent. In addition, the selection module 306 sends a credit usage signal 324 to the credit counter 302, indicating that a packet is being sent to the processing client 106. As described further below with reference to FIG. 4, in some embodiments, the selection module 306 further indicates that other communications (e.g., probes) are authorized to be sent to the processing client 106, and possibly sends the credit usage signal 324 in response to the authorization of the other communications.

[0032] Incoming requests are sent as packets 330 to packet memory 308. Packet memory 308 stores the packets until selection module 306 indicates via packet selection signal 332 that the packets be sent to packet memory 314 as packets 342. Similarly, packet memory 314 stores the packets until packet selection module 312 indicates via packet selection signal 344 that the packets be sent to an addressed portion of processing client 106 (e.g., a processor core, a memory device, etc.).

[0033] 4 is a block diagram illustrating management of probes addressing the processing client 106 of FIG. 1 in accordance with some embodiments. In the illustrated embodiment, the coherent agent 136 includes a probe memory 402, a probe selection module 404, a spoof module 406, the selection module 306 of FIG. 3, and a probe memory 408. The processing client 106 includes a probe memory 410, a client memory 412, and a probe memory 414. While the illustrated embodiment shows a particular configuration of components, other combinations of components are contemplated in various embodiments. Additionally, in some embodiments, additional components, such as buffers or repeaters, are contemplated but are not shown for ease of illustration.

[0034] The probe 428 is received from another device in the processing system 100 (e.g., the memory controller 108 or the processing client 104) and stored in the probe memory 408. As described above with reference to FIG. 3 , the selection module 306 receives a selection enable signal 326 indicating that the selection module 306 is authorized to select a communication (e.g., a packet or a probe) to send to the processing client 106. In the illustrated embodiment, in response to the selection enable signal 326, the selection module 306 indicates via a probe select signal 427 that the probe memory 408 is authorized to send the probe to the processing client 106 as a probe 430. In some embodiments, the selection module 306 specifies the probe to be sent. In other embodiments, the probe memory 408 is a queue, and the next probe is sent. The probe memory 408 sends the selected probe as probe 430 to the probe memory 414 and to the spoof module 406.

[0035] The probe memory 414 receives the probe 430 and sends it to the client memory 412 as a probe 436. The probe 436 is processed in the client memory 412 and a resulting probe response 434 is generated and sent to the probe memory 410. The probe memory 410 receives the probe response 434 and sends it to the probe selection module 404 as a probe response 432.

[0036] As described above with reference to FIG. 2 , in some embodiments, resetting the processing client 106 includes responding to some probes with spoofed probe responses. Accordingly, probe 430 is sent to spoof module 406. Spoof module 406 generates spoofed probe response 426 in response to probe 430 and sends spoofed probe response 426 to probe selection module 404. In the illustrated embodiment, spoofed probe response 426 indicates a cache miss at the processing client 106. However, in other embodiments, spoofed probe response 426 indicates something else, such as that the processing client 106 has been reset. In some embodiments, spoof module 406 generates a spoofed probe response in response to each received probe. In other embodiments, spoof module 406 is enabled as part of the reset process of the processing client 106.

[0037] The probe selection module 404 selects between the probe response 432 and the spoofed probe response based on a spoof indicator 420. In the illustrated embodiment, the spoof indicator 420 indicates whether probe response spoofing is enabled (e.g., as part of a reset process for the processing client 106). If probe response spoofing is enabled, the spoofed probe response 426 is selected and output as the selected probe response 424. Otherwise, the probe response 432 is selected and output as the selected probe response 424. The probe memory 402 receives the selected probe response 424 and outputs the probe response as the selected probe response 422 to the requesting device, the memory controller 108, or another device in the processing system 100.

[0038] In some embodiments, the above-described apparatus and techniques are implemented in a system including one or more integrated circuit (IC) devices (also called integrated circuit packages or microchips), such as the processing systems described with reference to FIGS. 1-4 . Electronic design automation (EDA) and computer-aided design (CAD) software tools can be used in the design and manufacture of these IC devices. These design tools are typically represented as one or more software programs. The one or more software programs include code executable by a computer system for operating the computer system to operate on code representing the circuits of one or more IC devices to perform at least a portion of a process for designing or adapting a manufacturing system for manufacturing the circuits. This code may include instructions, data, or a combination of instructions and data. The software instructions representing the design or manufacturing tools are typically stored in a computer-readable storage medium accessible to the computing system. Similarly, code representing one or more stages of the design or manufacture of the IC devices is stored in and accessed from the same or a different computer-readable storage medium.

[0039] In some embodiments, a computer-readable storage medium includes any non-transitory storage medium or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tape, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical systems (MEMS)-based storage media. The computer-readable storage medium (e.g., system RAM or ROM) may be internal to the computing system, the computer-readable storage medium (e.g., a magnetic hard drive) may be permanently attached to the computing system, the computer-readable storage medium (e.g., an optical disk or Universal Serial Bus (USB)-based flash memory) may be removably attached to the computing system, or the computer-readable storage medium (e.g., network-accessible storage (NAS)) may be coupled to the computer system via a wired or wireless network.

[0040] In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied in a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as flash memory, a cache, a random access memory (RAM), or other non-volatile memory device(s). The executable instructions stored on the non-transitory computer-readable storage medium may be implemented as source code, assembly language code, object code, or other form of instructions that can be interpreted or otherwise executed by one or more processors.

[0041] In addition to the above, it should be noted that not all activities or elements described in the summary description are required, that some of the particular activities or devices may not be required, that one or more additional activities may be performed, and that one or more additional elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and variations can be made without departing from the scope of the invention as set forth in the claims. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present invention.

[0042] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and features from which any benefit, advantage, or solution may arise or be manifested are not construed as critical, essential, or essential features of any or all claims. Moreover, the specific embodiments described above are illustrative only, since the disclosed invention may be modified and practiced in different, but similar manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the appended claims. It is therefore apparent that the specific embodiments described above may be altered or modified, and that all such variations are considered within the scope of the disclosed invention. Accordingly, the protection sought herein is set forth in the appended claims.

[0043] Within this disclosure, different entities (which may be variously referred to as "components," "units," "devices," etc.) may, at times, be described or claimed as being "configured" to perform one or more tasks or operations. This express phrase, an entity configured to "perform one or more tasks," is used herein to refer to a structure (i.e., a physical thing, such as an electronic circuit). More specifically, this express phrase is used to indicate that the structure is adapted to perform one or more tasks during operation. A structure may be said to be "configured to" perform some task even if the structure is not currently being operated. A "memory device configured to store data," for example, is intended to encompass an integrated circuit having circuitry that stores data during operation, even if the subject integrated circuit is not currently being used (e.g., no power source is connected to it). Thus, an entity described or listed as being "configured to" perform some task refers to something physical, such as a device, circuit, memory that stores executable program instructions to perform a task. This phrase is not used herein to refer to something intangible. Furthermore, the term "configured to" is not intended to mean "configurable to." For example, an unprogrammed field programmable gate array is not considered to be "configured to" perform any particular function, but may be "configurable to" perform that function after programming. Additionally, reciting in the appended claims that a structure is "configured to" perform one or more tasks is clearly not intended to be construed as having means-plus-function elements.

Claims

1. 1. A method comprising: a coherent agent of a data fabric resetting a first client processor of the processing system without resetting a second client processor of the processing system, the resetting being performed by instructing the first client processor to stop requests to one or more devices connected to a shared data fabric of the processing system; the coherent agent blocking status communication between the first client processor and a memory controller of the shared data fabric; and the coherent agent causing the first client processor to reset; the coherent agent re-enabling the first client processor; the first client processor includes a first set of processor cores, and the second client processor includes a second set of processor cores different from the first set of processor cores; method.

2. and ceasing the request by the first client processor includes the first client processor terminating a chained write operation and completing the write operation.

10. The method of claim 1.

3. and after blocking the status communication, indicating non-coherency by the first client processor to the memory controller. The method of claim 1 or 2.

4. Re-enabling the first client processor includes: resuming the status communication between the first client processor and the memory controller; indicating coherency of the first client processor to the memory controller; The method of claim 3.

5. Blocking the status communication causes the first client processor to enter a temporary offline state; resuming the status communication ends the temporary offline state of the first client processor. The method of claim 4.

6. and after the first client processor enters the temporary offline state, waiting for any remaining status requests to the first client processor to be responded to before indicating that the first client processor is non-coherent. The method of claim 5.

7. Blocking the status communication includes disabling flow control checks on status responses by the first client processor; resuming the status communication includes enabling flow control checks on status responses by the first client processor. The method of claim 4.

8. Blocking the status communication includes enabling spoofing of probes sent to the first client processor; resuming the status communication includes disabling spoofing of probes sent to the first client processor. The method of claim 4.

9. enabling probe spoofing includes transmitting, by a coherent agent associated with the first client processor, a spoofed probe response generated in response to a probe addressed to the first client processor; 9. The method of claim 8.

10. Enabling spoofing of a probe includes generating, by a coherent agent associated with the first client processor, the spoofed probe response.

10. The method of claim 9.

11. causing the first client processor to be reset includes instructing a coherent agent associated with the first client processor to treat the first client processor as having a fuse-off state in the shared data fabric. The method of any one of claims 1 to 10.

12. A device a parallel processing unit; a central processing unit (CPU); a data fabric; The data fabric comprises: A memory controller; a coherent agent; The memory controller sending status communications from the CPU to the parallel processing units, the status communications including packets and probes; The coherent agent: determining that the parallel processing units are reset without resetting the CPU; In response to determining that the parallel processing unit is to be reset, blocking the status communication between the memory controller and the parallel processing unit; triggering a reset of the parallel processing unit; configured to: device.

13. The memory controller and further configured to mark the parallel processing unit as non-coherent after status communication between the memory controller and the parallel processing unit has been blocked for at least a specified period of time. The device of claim 12.

14. The coherent agent: a credit counter configured to select between a first number of processing credits indicated by a credit available signal from said parallel processing unit and zero processing credits based on a credit reset signal; the credit reset signal is sent as part of blocking the status communication between the memory controller and the parallel processing unit.

14. A device according to claim 12 or 13.

15. The coherent agent: a selection module configured to instruct one or more memory devices of the coherent agent to send a status communication to the parallel processing unit based on an indication of processing credits of the parallel processing unit received from the credit counter.

15. The device of claim 14.

16. The coherent agent: a spoof module configured to send a spoofed probe response to the memory controller as part of blocking the status communication between the memory controller and the parallel processing unit. The device of claim 12.

17. 1. A method comprising: a coherent agent of a data fabric resetting a first client processor of a processing system comprising at least one graphics processing unit (GPU) and at least one central processing unit (CPU), wherein resetting the first client processor comprises the coherent agent intercepting status communications between the first client processor and a memory controller of a data fabric of the processing system, and the coherent agent causing the first client processor to be reset; the coherent agent re-enabling the first client processor. method.

18. processing credits of the first client processor are distributed to one or more devices connected to the data fabric; Intercepting the status communication comprises: wiping the processing credits of the first client processor; resetting the processing credits of the first client processor to an initial credit count; 18. The method of claim 17.

19. instructing the first client processor to stop requests to one or more devices connected to a data fabric of the processing system, including preventing processing credits of a second client processor from being allocated to the first client processor.

19. The method of claim 17 or 18.

20. intercepting the status communication includes disabling flow control checks on responses by the first client processor. The method of any one of claims 17 to 19.

Images