Shared data storage system with high availability and hot plugging

Abstract

Method comprising: in a first storage processor of a plurality of storage processors of a shared storage system, the first storage processor being coupled to a shared storage device having a plurality of storage devices: receiving a first connectivity verification request from an initiator device; transmitting a first ready response to the initiator device in response to the first connectivity verification request; and after transmitting the first readiness response: Detecting that the first storage processor is decoupled from the shared storage device; and in accordance with detecting that the first storage processor is decoupled from the shared storage device, transmitting a not-ready response to the initiator device in response to a second connectivity check request from the initiator device.

Description

TECHNICAL FIELD

[0001] The disclosed embodiments generally relate to shared storage systems, such as non-volatile memory systems, and more particularly to improved communications within the shared storage systems through high availability and hot-plugging support. BACKGROUND

[0002] Shared storage systems typically use one or more storage processors to manage storage operations performed on shared storage devices, such as storage operations requested by a client system. In a shared storage system that uses a single storage processor to manage a shared storage device, unreliable connectivity of the single storage processor will disrupt the operation of the shared storage system. Using multiple storage processors to manage the shared storage device improves connectivity but reduces communication speed and efficiency.Therefore, it would be desirable to have a system and method that improves both the connectivity and communication between components of a shared storage system for more stable operation of the shared storage system.

[0003] In this case, the disclosure of US 2012 / 0 124 312 A1, US 2016 / 0 110 270 A1, US 2016 / 0 179 637 A1 and US 2016 / 0 217 049 A1 may be helpful for understanding the present invention.

[0004] US 2012 / 0 124 312 A1 describes various systems and methods that can detect preferences and / or state transitions for asymmetric logical unit access (ALUA) to a logical unit (LUN) and use these preferences and / or transitions to control how a host accesses a LUN in an ALUA array. One such method includes detecting a preferred controller for a LUN and then determining that a current owner controller of the LUN is not the preferred controller. The method can then initiate a change of ownership from the current owner controller to the preferred controller. In another method, an initial state of a first controller with respect to a LUN is detected. The method then detects a subsequent state of the first controller with respect to the LUN after the initial state is detected.The method may then cause a computing device to access the LUN through a second controller if the subsequent state is not the active optimized state.

[0005] US 2016 / 0 110 270 A1 relates to an information processing device comprising a plurality of communication units and a processor. Each of the plurality of communication units has a plurality of connectors and is interchangeable. The processor is configured to execute a process that includes specifying an active port attached to one of the communication units to which a faulty active port is also attached, and switching the active port detected during the specifying to a standby port.

[0006] US 2016 / 0 179 637 A1 discloses a method for managing a secondary storage system. The method may include the secondary storage system performing a failover update of a status of at least one secondary path coupling a host computer to the secondary storage system. This causes the host computer to prioritize sending IO requests to the secondary storage system over sending the IO requests to a primary storage system. The failover update is triggered by a check performed by the secondary storage system, according to which (a) the secondary storage system is at an acceptable level of synchronization with the primary storage system when the check is performed, and (b) the primary storage system is disconnected from the host computer.The method further includes receiving IO requests from the host computer and responding to the IO requests by the secondary storage system.

[0007] Finally, US 2016 / 0 217 049 A1 describes methods, storage arrays, and computer-readable media for triggering failover between an active controller and a standby controller of a storage array. One method includes determining, by the active controller, a number of attached fabrics available between ports of the active controller and a host, and determining, by the standby controller, a number of attached fabrics available between the ports of the standby controller and the host. The method further includes executing a policy configuration to determine whether the number of attached fabrics for both the active controller and the standby controller reaches a failover trigger threshold.Failover is a forced failover where the active controller takes over the role of the standby controller and the standby controller takes over the role of the active controller. SUMMARY

[0008] Various embodiments of systems, methods, and apparatus within the scope of the appended claims each have different aspects, none of which are solely attributable to the attributes described in this document. Without limiting the scope of the appended claims, after consideration of this disclosure, and particularly after consideration of the "Detailed Description" section, it will be understood how aspects of various embodiments are used to enable high availability and hot plugging in shared storage systems.

[0009] The present invention is based on the object of enabling improved connectivity and communication within shared storage systems, such as non-volatile storage systems. This object is achieved by a method according to claim 1, shared storage systems according to any one of claims 12, 18, 21, and a non-volatile computer-readable storage medium according to claim 22. Advantageous embodiments may include features of dependent claims. In response to a first connectivity check request from an initiator device, a first storage processor transmits a first ready response to the initiator device.After transmitting the first ready response and detecting that it (the first storage processor) is decoupled from the shared storage device, the first storage processor transmits a not-ready response to the initiator device in response to a second connectivity check request from the initiator device. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order that the present disclosure may be understood in detail, a more particular description may be made with reference to the features of the various embodiments, some of which are illustrated in the accompanying drawings. However, the accompanying drawings merely illustrate the more pertinent features of the present disclosure and are therefore not to be considered limiting, as the description may admit of other effective features. Fig. 1 is a block diagram illustrating an implementation of a shared memory system according to some embodiments. Fig. 2 is a block diagram illustrating an implementation of a storage processor of a shared memory system according to some embodiments. Fig. 3A illustrates an example flowchart representation of a communication method in a shared storage system without hot-plugging support, according to some embodiments. Fig. 3B illustrates an exemplary flowchart representation of a method for enhanced communication in a shared memory system, according to some embodiments. Fig. 4 illustrates a conceptual flowchart representation of a method for enhanced communication in a shared memory system, according to some embodiments.

[0011] According to common practice, the various features illustrated in the drawings may not be drawn to scale. Consequently, the dimensions of the various features may be intentionally exaggerated or reduced for clarity. Furthermore, some of the drawings may not depict all components of a given system, method, or apparatus. Finally, like reference numerals may be used to refer to like features throughout the specification and the figures. DETAILED DESCRIPTION

[0012] The various implementations described herein include systems, methods, and / or devices used to enable high availability and hot plugging in shared storage systems. Some implementations include systems, methods, and / or devices to enhance connectivity and communications within shared storage systems.

[0013] (A1) More specifically, some embodiments include a method for operating in a shared storage system. In some embodiments, the method is executed in a first storage processor of a plurality of storage processors of the shared storage system. The first storage processor is coupled to a shared storage device having a plurality of storage devices. The method includes receiving a first connectivity check request from an initiator device and transmitting a first ready response to the initiator device in response to the first connectivity check request.The method also includes, after transmitting the first ready response, detecting that the first storage processor is decoupled from the shared storage device, and in accordance with detecting that the first storage processor is decoupled from the shared storage device, transmitting a not-ready response to the initiator device in response to a second connectivity check request from the initiator device.

[0014] (A2) In some embodiments of the method of A1, the first connectivity verification request is a test unit readiness request.

[0015] (A3) In some embodiments, the method of any one of A1-A2 further includes, in the first storage processor, after transmitting the unready response, detecting that the first storage processor is again coupled to the shared storage device, and in accordance with detecting that the first storage processor is again coupled to the shared storage device, transmitting a second ready response to the initiator device in response to a third connectivity check request from the initiator device.

[0016] (A4) In some embodiments of the method of any one of A1-A3, the initiator device transmits a third connectivity verification request to a second storage processor of the plurality of storage processors of the shared storage system in accordance with the first storage processor transmitting the unready response. The second storage processor is coupled to the shared storage device.In these embodiments, the method further includes, in the second storage processor, receiving the third connectivity verification request from the initiator device, transmitting a second ready response to the initiator device in response to the third connectivity verification request, and, after transmitting the second connectivity verification request to the initiator device, in accordance with a read request received from the initiator device, performing a read operation to read data from the shared storage device and transmitting the data read from the shared storage device to the initiator device.

[0017] (A5) In some embodiments, the method of any one of A1-A4 further includes, in the second storage processor, after transmitting the first ready response to the initiator device and before detecting that the first storage processor is decoupled from the shared storage device, according to a read request received from the initiator device, performing a read operation to read data from the shared storage device and transmitting the data read from the shared storage device to the initiator device.

[0018] (A6) In some embodiments of the method of any one of A1-A5, responses to connectivity verification requests received at the first storage processor are determined by the first storage processor according to a test unit readiness parameter stored by the first storage processor.

[0019] (A7) In some embodiments, the method of A6 includes, in the first storage processor, setting the test unit standby parameter to a non-standby value according to detecting that the first storage processor is decoupled from the shared storage device.

[0020] (A8) In some embodiments of the method of any one of A1-A7, communications between the plurality of storage processors and the shared storage device use SCSI (Small Computer System Interface) commands.

[0021] (A9) In some embodiments of the method of any one of A1-A8, connectivity verification requests are received from the initiator to the first storage processor according to an indication that the first storage processor is a preferred storage processor.

[0022] (A10) In some embodiments of the method of any one of A1-A9, the shared memory device comprises a non-volatile memory device.

[0023] (A11) In some embodiments of the method of any one of A1-A10, the shared memory device comprises a flash memory device.

[0024] (A12) In another aspect, a shared storage system includes a plurality of storage processors. A first storage processor of the plurality of storage processors is coupled to a shared storage device having a plurality of storage devices. The first storage processor is configured to receive a first connectivity check request from an initiator device and transmit a first ready response to the initiator device in response to the first connectivity check request.Additionally, the first storage processor is configured, after transmitting the first ready response, to detect that the first storage processor is decoupled from the shared storage device, and in accordance with detecting that the first storage processor is decoupled from the shared storage device, to transmit a not-ready response to the initiator device in response to a second connectivity check request from the initiator device.

[0025] (A13) In some embodiments of the shared memory system of A12, the first connectivity verification request is a test unit readiness request.

[0026] (A14) In some embodiments of the shared device according to any one of A12-A13, the first storage processor is configured, after transmitting the unready response, to detect that the first storage processor is again coupled to the shared storage device, and in accordance with detecting that the first storage processor is again coupled to the shared storage device, to transmit a second ready response to the initiator device in response to a third connectivity check request from the initiator device.

[0027] (A15) In some embodiments of the shared storage system of any one of A12-A14, the initiator device transmits a third connectivity check request to a second storage processor of the plurality of storage processors of the shared storage system in accordance with the first storage processor transmitting the unready response.In some embodiments, the second storage processor is coupled to the shared storage device, and the second storage processor is configured to receive the third connectivity verification request from the initiator device, transmit a second ready response to the initiator device in response to the third connectivity verification request, and, after transmitting the second connectivity verification request to the initiator device, perform a read operation to read data from the shared storage device in accordance with a read request received from the initiator device, and transmit the data read from the shared storage device to the initiator device.

[0028] (A16) In some embodiments of the shared storage system according to any one of A12-A15, the first storage processor is configured, after transmitting the first ready response to the initiator device and before detecting that the first storage processor is decoupled from the shared storage device, to perform a read operation to read data from the shared storage device according to a read request received from the initiator device and to transmit the data read from the shared storage device to the initiator device.

[0029] (A17) In some embodiments of the shared storage system of any one of A12-A16, responses to connectivity verification requests received at the first storage processor are determined by the first storage processor according to a test unit readiness parameter stored by the first storage processor.

[0030] (A18) In some embodiments of the shared memory system of A17, the first memory processor is configured to set the test unit readiness parameter to a non-readiness value in accordance with detecting that the first memory processor is decoupled from the shared memory device.

[0031] (A19) In some embodiments of the shared storage system of any one of A12-A18, communications between the plurality of storage processors and the shared storage device use SCSI (Small Computer System Interface) commands.

[0032] (A20) In some embodiments of the shared storage system of any one of A12-A19, connectivity verification requests are received from the initiator to the first storage processor according to an indication that the first storage processor is a preferred storage processor.

[0033] (A21) In some embodiments of the shared memory system of A12-A20, the shared memory device comprises a non-volatile memory device.

[0034] (A22) In some embodiments of the shared memory system of A12-A21, the shared memory device comprises a flash memory device.

[0035] (A23) In another aspect, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a shared memory system, the one or more programs including instructions that, when executed by the one or more processors, cause the shared memory system to perform the method of any one of A1-A11 described above.

[0036] (A24) In another aspect, a shared memory system includes one or more processors, memory, and one or more programs stored in the memory. The one or more programs include instructions for receiving a first connectivity check request from an initiator device and transmitting a first ready response to the initiator device in response to the first connectivity check request.The one or more programs further include instructions, after transmitting the first ready response, for detecting that the first storage processor is decoupled from the shared storage device, and in accordance with detecting that the first storage processor is decoupled from the shared storage device, for transmitting a not-ready response to the initiator device in response to a second connectivity check request from the initiator device.

[0037] (A25) In some embodiments of the shared storage system of A24, the one or more processors include one or more processors of a first storage processor of a plurality of storage processors. In some embodiments, the first storage processor is coupled to a shared storage device having a plurality of storage devices. Furthermore, in some embodiments, the one or more programs include a client interface module configured to receive one or more connectivity verification requests from an initiator device and configured to communicate one or more responses to the initiator device, and a shared storage interface module configured to detect whether the first storage processor is decoupled from the shared storage device.

[0038] (A26) In some embodiments of the shared memory system of any one of A24-25, the one or more programs further include instructions for performing the method of any one of A1-A11.

[0039] (A27) In yet another aspect, a shared storage system includes a first storage processor of a plurality of storage processors of the shared storage system. In some embodiments, the first storage processor is coupled to a shared storage device having a plurality of storage devices, and the first storage device includes: means for receiving a first connectivity check request from an initiator device; means for transmitting a first ready response to the initiator device in response to the first connectivity check request; and means activated after transmitting the first ready response, including: means for detecting that the first storage processor is uncoupled from the shared storage device;and means, activated in accordance with detecting that the first storage processor is decoupled from the shared storage device, for transmitting a not-ready response to the initiator device in response to a second connectivity check request from the initiator device;

[0040] (A28) In some embodiments, the shared memory system of A27 further comprises means for performing the method of any one of A1-A11.

[0041] Numerous details are described herein to facilitate a thorough understanding of the exemplary implementations illustrated in the accompanying drawings. However, some embodiments may be practiced without numerous specific details, and the scope of the claims is limited only by those features and aspects specifically recited in the claims. Furthermore, well-known methods, components, and circuits have not been described in detail so as not to unnecessarily obscure the clear presentation of more pertinent aspects of the implementations described herein.

[0042] Fig. 1A is a block diagram illustrating an implementation of a shared storage system 100 according to some embodiments. While some example features are illustrated, various other features have been omitted for the sake of brevity and to avoid unnecessarily distracting from the pertinent aspects of the embodiments described herein. To this end, as a non-limiting example, shared storage system 100 includes a plurality of processors, including storage processor 120-1 and storage processor 120-2. In some embodiments, the plurality of processors are coupled to a client system (sometimes referred to as a client, client device, initiator, or initiator device), such as initiator / client 110 (referred to herein as initiator 110).In some embodiments, initiator 110 includes one or more storage processor interfaces, such as storage processor interfaces 112-1 and 112-2, used for communications between initiator 110 and the plurality of processors 120. In some embodiments, the plurality of processors is coupled to a shared memory device, such as shared memory device 130. In some embodiments, shared memory device 130 includes a plurality of memory devices 134-1, 134-2 through 134n and one or more shared memory controllers, such as shared memory controllers 132-1 and 132-2, used for communications between the plurality of memory processors 120 and the plurality of memory devices 134.In some embodiments, shared storage device 130 includes one or more non-volatile memory devices (e.g., memory devices 134-1, 134-2 through 134 are non-volatile memory devices). In some embodiments, shared storage device 130 includes one or more flash memory devices (e.g., memory devices 134-1, 134-2 through 134 are flash memory devices).

[0043] In some embodiments, storage processor 120-1 includes a client interface 122-1 for communicating with a client device, such as initiator 110. In some embodiments, storage processor 120-1 is coupled to initiator 110 via client interface 122-1. In some embodiments, communications between storage processor 120-1 and initiator 110 use a defined interface standard for communication, such as a small computer systems interface (SCSI), an internet small computer systems interface (iSCSI), an InfiniBand (IB), Fibre Channel (FC), or other storage interface.

[0044] In some embodiments, storage processor 120-1 includes a shared storage interface 128-1 for communicating with a shared storage device, such as shared storage device 130. In some embodiments, storage processor 120-1 is coupled to shared storage device 130 via shared storage interface 128-1. In some embodiments, communications between storage processor 120-1 and shared storage device 130 use a defined interface standard for communication, such as a Small Computer Systems Interface (SCSI), Serial Advanced Technology Attachment (SATA), Serial Attached SCSI (SAS), or other storage interface.

[0045] In some embodiments, the storage processor 120-1 includes a hot-plug management module 126-1. In some embodiments, the hot-plug management module 126-1 monitors connectivity between the storage processor 120-1 and the shared storage device 130. In some embodiments, the storage processor 120-1 includes one or more processing units 124-1 (occasionally referred to herein as CPUs, processors, or hardware processors, and sometimes implemented using microprocessors, microcontrollers, or the like) configured to execute instructions in one or more programs (e.g., in the storage processor 120-1 or a component of the storage processor 120, such as the hot-plug management module 126-1).In some embodiments, one or more components share one or more CPUs 124-1 within the scope of, and in some cases beyond, the functionality of storage processor 120-1. In some embodiments, CPUs 124-1 manage communication to and from storage processor 120-1 via client interface 122-1 and / or shared memory interface 128-1.

[0046] In some embodiments, storage processor 120-2 includes similar elements as those described above with reference to storage processor 120-1. Storage processors 120-1 and / or 120-2 may include various additional features that have not been illustrated for brevity and to avoid distracting from pertinent features of the embodiments disclosed herein, and various arrangements of features may be possible.

[0047] Fig. Figure 2 is a block diagram illustrating an implementation of a storage processor 120-1 or 120-2 (hereinafter, storage processor 120) according to some embodiments. The processor 120 typically includes a memory or multiple processing units 124-1 for executing modules, programs, and / or instructions stored in memory 206 and thereby performing processing operations; memory 206 (sometimes referred to herein as controller memory); and one or more communication buses 208 for interconnecting these components. The communication buses 208 optionally include circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between them. In some embodiments, such as those described in Fig. 1, the storage processor 120 is coupled to the initiator 110 and the shared memory device 130 via communication buses 208. It should be noted that the Fig. 2 may vary depending on the configuration of a particular shared storage system and that the components shown in Fig. 2 shown representations, which according to Fig. 1 are merely non-limiting examples. Memory 206 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 206 may include one or more storage devices located remotely from processor(s) 124-1. In some embodiments, memory 206, or alternatively, the non-volatile storage device(s) in memory 206, include a non-volatile computer-readable storage medium.In some embodiments, memory 206, or the computer-readable storage medium of memory 206, stores the following programs, modules, and data structures, or a subset or superset thereof:. • Operating system 210, which includes procedures for handling various basic system services and for performing hardware-dependent tasks; • Client interface 122-1, used to communicate with other components, such as initiator 110; in some embodiments, client interface 122-1 includes a SCSI target module 218 that receives requests from the initiator(s) and sends responses to requests back to the initiator(s); • Hotplug management module 126-1, which is used to monitor connectivity with other components, such as the shared storage device 130; • Configuration information 212, which is used to store information used to set up the storage processor 120 and may include: ◯ Test unit readiness parameter 214, which is used to indicate a status of the storage processor 120, such as whether the storage processor 120 is ready to process commands from the initiator 110; and • shared storage devices 128-1, used to communicate with other components, such as shared storage device 130; in some embodiments, shared storage interface 128-1 includes a SCSI initiator module 220 that manages communications with shared storage device 130.

[0048] Each of the elements identified above may be stored in one or more of the above-listed storage devices that collectively comprise memory 206, and corresponds to a set of instructions for performing a function described above. The modules or programs (i.e., sets of instructions) identified above need not be implemented as separate software programs, methods, or modules, and thus various subsets of these modules may be combined or otherwise rearranged in various embodiments. In some embodiments, memory 206 may store a subset of the modules and data structures identified above. Furthermore, memory 206 may store additional modules and data structures not described above.In some embodiments, the programs, modules, and data structures stored in memory 206 or the computer-readable storage medium of memory 206 provide instructions for implementing corresponding operations in the embodiments described below with reference to FIG. Fig. 3A-3B and 4.

[0049] Although Fig. 2 shows the storage processor 120-1, Fig. 2 is to be understood as a functional description of the various features that may be present in the storage processor, rather than as a structural diagram of the embodiments described herein. In practice, and as will be appreciated by those skilled in the art, the separately illustrated elements could be combined and individual elements separated. Moreover, as mentioned above, in some embodiments, one or more modules of storage processor 120-1 are implemented in storage processor 120-1 or storage processor 120-2 of shared storage system 100. Fig. 1.

[0050] Fig. 3A illustrates an exemplary flowchart representation of a communication method 300 in a shared storage system without hot-plugging support, according to some embodiments. For ease of explanation, the following describes the method 300 as performed by a first storage processor A (e.g., storage processor 120-1, Fig. 1), a second storage processor B (e.g., storage processor 120-2, Fig. 1) and an initiator (e.g. initiator 110, Fig. 1) operatively coupled to storage processor A and storage processor B, wherein communications between the storage processors and the initiator use SCSI (Small Computer Systems Interface) commands. The method begins with storage processor A (e.g., storage processor 120-1, Fig. 1) and storage processor B (e.g., storage processor 120-2) are both connected to a shared storage device (e.g., shared storage device 130, Fig. 1), with communications between the storage processors and the shared storage device also using SCSI commands.

[0051] Storage processor A receives (302) a test-unit-ready (TUR) request from the initiator. In response to the test-unit-ready request, storage processor A transmits (304) a ready response to the initiator.

[0052] After transmitting the ready response (304), storage processor A receives (306) a Report Target Port Groups (RTPG) request from the initiator. In response, storage processor A transmits (308) a preference indicator (PREF) bit with a value set to "1," indicating that storage processor A is a preferred storage processor for accessing the shared storage device.

[0053] After transmitting the indication that it is a preferred storage processor, storage processor A receives (310) a read request from the initiator to read data from the shared storage device. In response, storage processor A reads data from the shared storage device and transmits (312) the data read from the shared storage device to the initiator.

[0054] Next (314) storage processor A is disconnected from the shared storage device so that storage processor A loses communication with the shared storage device.

[0055] While storage processor A is disconnected from the shared storage device, storage processor A transmits (318) a read error in response to a subsequent read request from the initiator to read data from the shared storage device (316) because storage processor A is unable to communicate with the shared storage device to read the requested data.

[0056] In response to the read error from storage processor A, the initiator performs (320) a failover operation, switching to storage processor B instead of storage processor A to access the shared storage device. Accordingly, the initiator transmits (322) a read request to storage processor B. In response, storage processor B reads data from the shared storage device and transmits (324) the data read from the shared storage device to the initiator.

[0057] Subsequently, the initiator transmits (326) a second TUR request to storage processor A. In response, storage processor A sends (328) a ready response to the initiator, even though storage processor A is still disconnected from the shared storage device. Accordingly, the initiator transmits (330) a second RTPG request to storage processor A, and in response, storage processor A (again) sends the PREF bit with a value set to "1," indicating that storage processor A is the preferred storage processor.

[0058] If storage processor A transmits a ready response and the PREF bit indicates that storage processor A is the preferred storage processor, the initiator performs (334) a failback operation to switch back to storage processor A to access the shared storage device. Accordingly, the initiator transmits (336) a read request to storage processor A. However, since storage processor A is still disconnected from the shared storage device and is unable to communicate with the shared storage device to read the requested data, storage processor A again transmits (338) a read miss.

[0059] In response to the read errors from storage processor A, the initiator again performs (340) a failover operation, switching to storage processor B instead to access the shared storage device, and accordingly transmits (342) a read request to storage processor B. In response, storage processor B reads data from the shared storage device and transmits (344) the data read from the shared storage device to the initiator.

[0060] Fig. 3B illustrates an exemplary flowchart representation of a method 301 for enhanced communication in a shared memory system, according to some embodiments. Referring to the Fig. 1, in some embodiments, some of the operations (or alternatively, steps) of method 301 are performed at least partially in a storage processor (e.g., storage processor 120-1, Fig. 1) or in one or more components of the storage processor (e.g., client interface 122-1, hotplug management module 126-1, and / or shared memory interface 128-1, Fig. 1). In some embodiments, method 301 is governed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as the one or more processing units (CPUs) 124-1 of storage processor 120-1 ( Fig. 1), are executed.

[0061] In some embodiments, some of the operations of method 301 are performed at an initiator (e.g., initiator 110, Fig. 1) operatively coupled to the storage processor, and / or at a shared storage device (e.g., shared storage device 130, Fig. 1) operatively coupled to the storage processor, and other operations of method 301 are performed at the storage processor. In some embodiments, method 301 is governed at least in part by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors (e.g., hardware processors) of the initiator and / or the shared storage device.

[0062] With reference to Fig. 2, in some embodiments, some of the operations of method 301 are performed at least in part by a hotplug management module (e.g., hotplug management module 126-1, Fig. 2) and in some cases using a client interface (e.g. client interface 122-1, Fig. 2) and a shared memory interface (e.g. shared memory interface 128-1, Fig. 2) of a respective storage processor.

[0063] For ease of explanation, the following describes method 301 as performed by a particular storage processor (e.g., storage processor 120-1, Fig. 1) and an initiator (e.g. initiator 110, Fig. 1), which is operatively coupled to the storage processor. In the Fig. 3B, communications between storage processor 120-1 and initiator 110 use SCSI (Small Computer Systems Interface) commands. Accordingly, the storage processor is connected to the initiator via a SCSI target module 218 (or more generally, a client interface, such as client interface 122-1). Fig. 1) is coupled to the storage processor 120 used for communications with the initiator (e.g., storage processor 230 is a SCSI target with respect to initiator 110). In some embodiments, initiator 110 similarly includes an interface (e.g., storage processor interface 112-1, Fig. 1) used to communicate with storage processor 120. It should be understood that method 310 is also performed on the other storage processor(s) of the shared storage system when requests from the initiator are directed to those other storage processors. In some embodiments, other communication protocols different from those described herein are used in performing method 301.

[0064] The method 301 begins with the storage processor (e.g., storage processor 120-1, Fig. 1) that is also operatively connected to a shared storage device (e.g., the shared storage device 130, Fig. 1) is coupled. In the Fig. 3B, the storage processor is connected to the shared storage device through a SCSI initiator module (or more generally, a shared storage interface, such as shared storage interface 128-1, Fig. 1) coupled to the storage processor used for communications with the shared storage device (e.g., the storage processor is a SCSI initiator with respect to the shared storage device). Additionally, the storage processor includes a hotplug management module for managing connectivity (e.g., between the storage processor and the shared storage device). In some embodiments, the shared storage device similarly includes an interface (e.g., the shared storage controller 132-1, Fig. 1) which is used to communicate with the storage processor.

[0065] The storage processor (e.g., SCSI target module 218) receives (350) a test unit ready (TUR) request from initiator 110. In response to the test unit ready request, the storage processor transmits (352) a ready response to the initiator (e.g., to indicate that the storage processor is available to process commands from the initiator to access data stored on the shared storage device). In some embodiments, the storage processor (or a component of the storage processor, such as SCSI target module 218) stores configuration information (e.g., configuration information 212, Fig. 2) that include a test unit readiness parameter (e.g., test unit readiness parameter 214, Fig. 2). In some embodiments, the test unit readiness parameter indicates whether the storage processor is in a ready state (e.g., coupled to the shared storage device) or a non-ready state (e.g., uncoupled from the shared storage device). In some embodiments, the test unit readiness parameter is updated according to the determination of whether the storage processor is coupled to the shared storage device (e.g., the test unit readiness parameter is set or updated to a ready value according to the determination that the storage processor is or has been coupled to the shared storage device, or set or updated to a non-ready value according to the determination that the storage processor is or has been uncoupled from the shared storage device).In some embodiments, the storage processor determines responses to test unit readiness requests from the initiator according to the test unit readiness parameter stored in the storage processor.

[0066] After (or in response to) transmitting the ready response (352), in some embodiments, storage processor A receives (354) a Report Target Port Groups (RTPG) request from the initiator. In response to the RTPG request from the initiator, the storage processor transmits (356) an RTPG response. In some embodiments, the RTPG response transmitted by the storage processor includes a preference indicator (PREF) bit with a value set to "1," indicating that the storage processor is a preferred storage processor for accessing data stored on the shared storage device.

[0067] After transmitting the RTPG response (e.g., indicating that it is a preferred storage processor), the storage processor (e.g., the SCSI target module) receives (358) a read request from the initiator to read data from the shared storage device (e.g., the request is received before the storage processor detects that it is disconnected from the shared storage device). In some embodiments, the read request from the initiator is received by the storage processor's SCSI target module, and the read request is transmitted (360) to the SCSI initiator module 220 or the shared storage interface 128-1 used for communications with the shared storage device.In response (and while the storage processor is connected to the shared storage device), the storage processor performs (362) a read operation to read data from the shared storage device (e.g., the storage processor uses the SCSI initiator module 220 to read data from the shared storage device). The storage processor (e.g., the SCSI initiator module 220) receives (364) the data read from the shared storage device and transmits (e.g., using the SCSI target module 218 (366)) the data to the initiator (368).

[0068] Next, the storage processor (370) is disconnected from the shared storage device (e.g., such that the storage processor loses communication with the shared storage device). The storage processor detects that the shared storage device is disconnected (e.g., the shared storage device is removed). For example, the SCSI initiator module 220 detects that the shared storage is disconnected and reports (372) the disconnection to the hotplug management module 126-1. In response, the hotplug management module modifies (374) the storage processor's response to test unit readiness requests. In some embodiments, the hotplug management module sets or updates the test unit readiness parameter (e.g., test unit readiness parameter 214 stored in the configuration information 212, Fig. 2) to indicate that the storage processor is in a non-ready state.

[0069] While the storage processor is disconnected from the shared storage device, the storage processor transmits (376) a not-ready response (e.g., according to the test unit ready parameter set to a not-ready value by the storage processor in response to detecting the disconnection from the shared storage device, as discussed above with reference to operation 374) in response to a subsequent test unit ready request from the initiator (378).

[0070] According to the storage processor transmitting the unready response, the initiator is notified that the storage processor is unable to process commands from the initiator (e.g., requests to perform memory operations, such as read and / or write commands) to access the shared storage device. In some embodiments, the initiator subsequently does not transmit an RTPG request. In some embodiments, in response to receiving the unready response from the (first) storage processor, the initiator transitions to a second storage processor (e.g., storage processor 120-2, Fig. 1, or storage processor B, Fig. 3A) for accessing the shared storage device, as described in further detail herein with reference to Fig. 4 described.

[0071] Next, (380) the (first) storage processor is reconnected to the shared storage device (e.g., restoring communication between the storage processor and the shared storage device). The storage processor detects that the shared storage device is reconnected. For example, the SCSI initiator module 220 detects that the shared storage is reconnected and reports (382) the reconnection to the hotplug management module 126-1. In response, the hotplug management module modifies (384) the storage processor's response to (subsequently received) test unit readiness requests. In some embodiments, the hotplug management module updates the test unit readiness parameter (e.g., test unit readiness parameter 214 stored in the configuration information 212, Fig. 2) to indicate that the storage processor is (again) in a ready state.

[0072] After the storage processor is reconnected to the shared storage device, the storage processor transmits (386) a ready response in response to a subsequent test unit ready request from the initiator (388) (e.g., according to the test unit ready parameter updated by the storage processor in response to detecting the reconnection to the shared storage device, as discussed above with reference to operation 384).

[0073] After transmitting the ready response (388), the storage processor receives (390) an RTPG request from the initiator. In response to the RTPG request from the initiator, the storage processor transmits (392) an RTPG response, which in some embodiments includes a PREF bit with a value set to "1," indicating that the storage processor is a preferred storage processor for accessing the shared storage device. In some embodiments, the PREF bit is independent of whether the storage processor is connected to a shared storage device (e.g., the value of the PREF bit is maintained regardless of whether the storage processor is connected to or disconnected from a shared storage device).

[0074] After transmitting the RTPG response (e.g., indicating that it is a preferred storage processor), the storage processor receives (394) a read request from the initiator to read data from the shared storage device. In some embodiments, in response to the read request, method 301 continues as described above with reference to operations 360, 362, 364, 366, and 368. For brevity, these details are neither illustrated nor repeated here.

[0075] Fig. 4 illustrates a conceptual flowchart representation of a method 400 for enhanced communication in a shared memory system, according to some embodiments. Referring to the Fig. 1, in some embodiments, some of the operations (or alternatively, steps) of method 400 are performed at least partially in a first storage processor (e.g., storage processor 120-1, Fig. 1) or in one or more components of the first storage processor (e.g., client interface 122-1, hotplug management module 126-1, and / or shared memory interface 128-1, Fig. 1). In some embodiments, method 400 is governed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as the one or more processing units (CPUs) 124-1 of storage processor 120-1 ( Fig. 1). In some embodiments, some of the operations of method 400 are performed at least partially in a second storage processor (e.g., storage processor 120-2, Fig. 1) or in one or more components of the second storage processor (e.g., client interface 122-2, hotplug management module 126-2, and / or shared memory interface 128-2, Fig. 1). In some embodiments, method 400 is governed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as the one or more processing units (CPUs) 124-2 of storage processor 120-2 ( Fig. 1), are executed.

[0076] In some embodiments, some of the operations of method 400 are performed at an initiator (e.g., initiator 110, Fig. 1) operatively coupled to the first and second storage processors, and / or at a shared storage device (e.g., shared storage device 130, Fig. 1) operatively coupled to the first and second storage processors, and other operations of method 301 are performed at a respective storage processor. In some embodiments, method 400 is governed at least in part by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors (e.g., hardware processors) of the initiator and / or the shared storage device.

[0077] With reference to Fig. 2, in some embodiments, some of the operations of method 400 are performed at least in part by a hotplug management module (e.g., hotplug management module 126-1, 2) and, in some cases, using a client interface (e.g., client interface 122-1, 2) and a shared memory interface (e.g., shared memory interface 128-1, Fig. 2) of each one.

[0078] For ease of explanation, the following describes the method 400 as it occurs at an initiator (e.g., initiator 110, Fig. 1) and a plurality of storage processors, including a first storage processor A (e.g., storage processor 120-1, Fig. 1) and a second storage processor B (e.g. storage processor 120-2, Fig. 1). The method begins with the storage processor (e.g., storage processor 120-1, Fig. 1) that is connected to a shared storage device (e.g., the shared storage device 130, Fig. 1) is coupled (e.g., connected). In some embodiments, communications between the plurality of storage processors and the shared storage device use SCSI (Small Computer System Interface) commands (e.g., as described above with reference to method 301, Fig. 3B). The initiator transmits (402) a (first) connectivity check request (e.g., a test unit ready request, for example, a test unit ready request conforming to SCSI standards) to storage processor A. Storage processor A receives (404) the connectivity check request. In response to the connectivity check request and, in some embodiments, in accordance with a determination that storage processor A is in a ready state, storage processor A transmits (406) a ready response to the initiator (e.g., to indicate that storage processor A is available to process commands from the initiator to access data stored on the shared storage device).

[0079] In some embodiments, storage processor A stores configuration information (e.g., configuration information 212, Fig. 2) that include a test unit readiness parameter (e.g., test unit readiness parameter 214, Fig. 2). As described above with reference to operation 352 of method 301 ( Fig. 3B), in some embodiments, the test unit readiness parameter indicates whether the storage processor is in a ready state or a non-ready state, and the test unit readiness parameter is updated according to the determination of whether the storage processor is coupled to the shared storage device. In some embodiments, storage processor A determines responses to connectivity check requests (e.g., test unit readiness requests) from the initiator according to the test unit readiness parameter stored in storage processor A.

[0080] In some embodiments, after transmitting the ready response to the initiator (406), storage processor A receives a request from the initiator for additional configuration information. In some embodiments, in response to the request for additional configuration information, storage processor A transmits an indication that it (storage processor A) is a preferred storage processor (e.g., a preferred storage processor for accessing data stored on the shared storage device). For example, in some embodiments, storage processor A stores configuration information (e.g., configuration information 212, Fig. 2) that use a preference indicator (e.g. preference indicator 216, Fig. 2) indicating whether storage processor A is a preferred storage processor (e.g., the preference indicator is a bit, where a bit value of "1" indicates that the storage processor is a preferred storage processor, while a bit value of "0" indicates that the storage processor is not a preferred storage processor). In some embodiments, processor A receives a Report Target Port Groups (RTPG) request from the initiator and, in response, transmits a preference indicator (PREF) bit with a value set to "1" indicating that storage processor A is a preferred storage processor (e.g., according to preference indicator 216, Fig. 2).

[0081] In some embodiments, after storage processor A transmits (406) the ready response to the initiator (and in some embodiments, after storage processor A receives a request from the initiator for additional configuration information), the initiator transmits (408) a read request to storage processor A to read data from the shared storage device (or more generally, the initiator transmits a command or request to storage processor A to perform a store operation on the shared storage device).In some embodiments, if storage processor A receives the read request from the initiator while storage processor A is still coupled to the shared storage device (and thus before detecting that storage processor A is uncoupled from the shared storage device), storage processor A reads (410) the data from the shared storage device (or, more generally, storage processor A performs the requested store operation). The initiator, in turn, receives (412) the data transmitted by storage processor A.

[0082] In some embodiments or under certain circumstances, storage processor A detects (413) that it is decoupled (e.g., disconnected) from the shared storage device (e.g., as described in more detail above with reference to operations 370, 372, and 374 of method 301, Fig. 3B).

[0083] In some embodiments, while storage processor A is decoupled from the shared storage device, the initiator transmits (414) a second connectivity verification request (e.g., a second test unit ready request). In response, storage processor A transmits (416) an unready response (e.g., in accordance with storage processor A detecting that it is decoupled from the shared storage device, indicating that storage processor A is unavailable to process commands from the initiator to access data stored on the shared storage device).

[0084] In some embodiments, in response to receiving the unready response from storage processor A, the initiator switches to a second storage processor B for access to the shared storage device. In some embodiments, the initiator transmits (418) a third connectivity check request (e.g., a third test unit ready request) to storage processor B. In some embodiments, the initiator periodically transmits connectivity check requests to (e.g., polling) one or more of the plurality of storage processors to determine whether a respective storage processor is in a ready state (e.g., available to process commands from the initiator to access data stored on the shared storage device).For example, in some embodiments, the initiator periodically transmits connectivity check requests to storage processor B (or to both storage processor A and storage processor B), regardless of whether the storage processor is in a ready state. In some embodiments (e.g., in accordance with a determination that storage processor B is coupled to the shared storage device), storage processor B transmits (420) a ready response to the initiator in response to the third connectivity check request (e.g., storage processor B is available to process commands from the initiator to access data stored on the shared storage device).

[0085] In some embodiments, in response to receiving the not-ready response from storage processor A (416) (and in some embodiments, in response to receiving the ready response from storage processor B (420)), the initiator transmits (422) a read request to storage processor B to read data from the shared storage device. In response, storage processor B reads (424) the data from the shared storage device and transmits the read data to the initiator. The initiator, in turn, receives (426) the data transmitted from storage processor B.

[0086] In some embodiments or under certain circumstances, storage processor A detects (427) that it is re-coupled (e.g., reconnected) to the shared storage device (e.g., as described in more detail above with reference to operations 380, 382, and 384 of method 301, Fig. 3B).

[0087] In some embodiments, after storage processor A is recoupled to the shared storage device, the initiator transmits (428) a fourth connectivity check request to storage processor A, and in response, storage processor A transmits (430) a ready response to the initiator (e.g., to indicate that storage processor A is available to process commands from the initiator to access data stored on the shared storage device). In some embodiments, storage processor A receives connectivity check requests from the initiator according to an indication that storage processor A is a preferred storage processor (e.g., as described in more detail above with reference to operations 354, 356, 390, and 392 of method 301). Fig. 3B). In some embodiments, after the initiator switches to storage processor B (e.g., according to the not-ready response from storage processor A) and requests storage processor B to perform one or more stores on the shared storage device, the initiator switches back to storage processor A before requesting the performance of a subsequent store or more subsequent stores on the shared storage device, in accordance with the indication that storage processor A is a preferred storage processor.

[0088] It should be noted that the procedure 301 of Fig. 3B and the method 400 of Fig. 4 bypass the failback and failover operations described herein with reference to the method 300 of Fig.3A, thereby improving the speed and efficiency of communications between the plurality of processors, the initiator, and the shared memory system, and increasing the robustness of the shared memory system.

[0089] It should be understood that although the terms "first," "second," etc., may be used herein to describe various elements, these elements are not intended to be limited to these terms. These terms are used only to distinguish one element from another. For example, a first contact could be referred to as a second contact, and similarly, a second contact could be referred to as a first contact, changing the meaning of the description, provided that each occurrence of "first contact" is systematically relabeled and each occurrence of "second contact" is systematically relabeled. The first contact and the second contact are both contacts, but they are not the same contact.

[0090] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of the embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It is also understood that the term "and / or," as used herein, refers to any and all possible combinations of one or more of the associated listed elements.It is further understood that the terms "comprise" and / or "comprising" when used in this specification denote the presence of specified features, units, steps, acts, elements and / or components, but do not preclude the presence or addition of one or more other features, units, steps, acts, elements, components and / or groups thereof.

[0091] As used herein, the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "according to a determination" or "in response to the determination" that a stated prerequisite is met, depending on the textual context. Similarly, the phrase "when it is determined [that a stated prerequisite is met]" or "if [a stated prerequisite is met]" or "when [a stated prerequisite is met]" may be interpreted to mean: "upon determining" or "in response to the determination" or "according to a determination" or "upon recognizing" or "in response to recognizing" that an stated prerequisite is met, depending on the textual context.

[0092] The foregoing description has been provided for illustrative purposes with reference to specific implementations. However, the foregoing illustrative discussions are neither exhaustive nor intended to limit the claims to the precise forms disclosed. Numerous modifications and variations are possible in light of the above teachings. The implementations were chosen and described to best explain the principles of operation and practical applications, thereby enabling others skilled in the art to appreciate the invention.

Claims

[1] Method comprising: in a first storage processor of a plurality of storage processors of a shared storage system, the first storage processor being coupled to a shared storage device having a plurality of storage devices: receiving a first connectivity verification request from an initiator device; transmitting a first ready response to the initiator device in response to the first connectivity verification request; and after transmitting the first readiness response: Detecting that the first storage processor is decoupled from the shared storage device; and in accordance with detecting that the first storage processor is decoupled from the shared storage device, transmitting a not-ready response to the initiator device in response to a second connectivity check request from the initiator device. [2] The method of claim 1, wherein the first connectivity verification request is a test unit readiness request. [3] Method according to one of claims 1-2, further comprising: in the first storage processor, after transmitting the non-ready response: Detecting that the first storage processor is again coupled to the shared storage device; and, in accordance with detecting that the first storage processor is again coupled to the shared storage device, transmitting a second ready response to the initiator device in response to a third connectivity verification request from the initiator device. [4] The method of any of claims 1-3, wherein the initiator device transmits a third connectivity check request to a second storage processor of the plurality of storage processors of the shared storage system in accordance with the first storage processor transmitting the unready response, the second storage processor being coupled to the shared storage device, the method further comprising: in the second storage processor: Receiving the third connectivity verification request from the initiator device; transmitting a second ready response to the initiator device in response to the third connectivity verification request; and, after transmitting the second connectivity verification request to the initiator device, according to a read request received from the initiator device, performing a read operation to read data from the shared storage device, and transmitting the data read from the shared storage device to the initiator device. [5] Method according to one of claims 1-4, further comprising: at the first storage processor, after transmitting the first ready response to the initiator device and before detecting that the first storage processor is decoupled from the shared storage device: according to a read request received from the initiator device, performing a read operation to read data from the shared storage device, and transmitting the data read from the shared storage device to the initiator device. [6] The method of any of claims 1-5, wherein responses to connectivity verification requests received at the first storage processor are determined by the first storage processor according to a test unit readiness parameter stored by the first storage processor. [7] The method of claim 6, further comprising, at the first storage processor, setting the test unit readiness parameter to a non-ready value in accordance with detecting that the first storage processor is decoupled from the shared storage device. [8] The method of any of claims 1-7, wherein communications between the plurality of storage processors and the shared storage device use SCSI (Small Computer System Interface) commands. [9] The method of any of claims 1-8, wherein connectivity verification requests are received from the initiator to the first storage processor in accordance with an indication that the first storage processor is a preferred storage processor. [10] The method of any of claims 1-9, wherein the shared storage device comprises a non-volatile storage device. [11] The method of any of claims 1-10, wherein the shared memory device comprises a flash memory device. [12] Shared storage system comprising: a plurality of processors, wherein a first storage processor of the plurality of storage processors is coupled to a shared storage device having a plurality of storage devices, and wherein the first storage processor is configured to: receiving a first connectivity verification request from an initiator device; transmitting a first ready response to the initiator device in response to the first connectivity verification request; and after transmitting the first readiness response: Detecting that the first storage processor is decoupled from the shared storage device; and in accordance with detecting that the first storage processor is decoupled from the shared storage device, transmitting a not-ready response to the initiator device in response to a second connectivity check request from the initiator device. [13] The shared storage system of claim 12, wherein the first connectivity verification request is a test unit readiness request. [14] Shared storage system according to one of claims 12-13, wherein the first storage processor is further configured to: after transmitting the non-readiness response: Detecting that the first storage processor is again coupled to the shared storage device; and, in accordance with detecting that the first storage processor is again coupled to the shared storage device, transmitting a second ready response to the initiator device in response to a third connectivity verification request from the initiator device. [15] Shared storage system according to any one of claims 12-14, wherein the initiator device transmits a third connectivity check request to a second storage processor of the plurality of storage processors of the shared storage system in accordance with the transmission of the unready response by the first storage processor, the second storage processor being coupled to the shared storage device, the shared storage processor being configured to: Receiving the third connectivity verification request from the initiator device; transmitting a second ready response to the initiator device in response to the third connectivity verification request; and, after transmitting the second connectivity verification request to the initiator device, according to a read request received from the initiator device, performing a read operation to read data from the shared storage device, and transmitting the data read from the shared storage device to the initiator device. [16] Shared storage system according to any one of claims 12-15, wherein the first storage processor is further configured to: after transmitting the first ready response to the initiator device and before detecting that the first storage processor is decoupled from the shared storage device: according to a read request received from the initiator device, performing a read operation to read data from the shared storage device, and transmitting the data read from the shared storage device to the initiator device. [17] The shared storage system of any of claims 12-16, wherein responses to connectivity verification requests received at the first storage processor are determined by the first storage processor according to a test unit readiness parameter stored by the first storage processor. [18] Shared storage system comprising: one or more processors; memory; and one or more programs stored in the memory, the one or more programs being adapted for execution by one or more processors and including instructions for: receiving a first connectivity verification request from an initiator device; transmitting a first ready response to the initiator device in response to the first connectivity verification request; and after transmitting the first readiness response: Detecting that the first storage processor is decoupled from the shared storage device; and, in accordance with detecting that the first storage processor is decoupled from the shared storage device, transmitting a not-ready response to the initiator device in response to a second connectivity check request from the initiator device. [19] A shared storage system according to claim 18, wherein: the one or more processors include one or more processors of the first storage processor of a plurality of storage processors; the first storage processor is coupled to a shared storage device having a plurality of storage devices; and the one or more programs include a client interface module configured to receive one or more connectivity verification requests from an initiator device and configured to transmit one or more responses to the initiator device, and a shared storage interface module configured to detect whether the first storage processor is decoupled from the shared storage device. [20] A shared memory system according to any one of claims 18-19, wherein the one or more programs further include instructions for performing the method according to any one of claims 1-11. [21] Shared storage system comprising: a first storage processor of a plurality of storage processors of a shared storage system, the first storage processor coupled to a shared storage device having a plurality of storage devices, and the first storage processor including: means for receiving a first connectivity verification request from an initiator device; means for transmitting a first ready response to the initiator device in response to the first connectivity verification request; and Means activated after the first readiness response is transmitted, including: means for detecting that the first storage processor is decoupled from the shared storage device; and Means, activated in accordance with detecting that the first storage processor is decoupled from the shared storage device, for transmitting a not-ready response to the initiator device in response to a second connectivity check request from the initiator device. [22] Shared storage system according to any one of claims 21, further comprising means for carrying out the method according to any one of claims 1-11. [23] A non-transitory computer-readable storage medium storing one or more programs adapted to be executed by one or more processors of a shared memory system, the one or more programs including instructions which, when executed by the one or more processors, cause the shared memory system to perform the method of any one of claims 1-11.

Images