Systems and methods for multiplexed multi-driver shoot-through commands

Abstract

The invention relates to a system and a method for multiplexing multi-driver shoot-through commands. The system may include one or more processors configured to receive a frame including a number of commands, a number of storage devices, and a buffer map. In response to the frame, the one or more processors may map and read (1) input data corresponding to each of the number of one or more storage devices and (2) an identifier of each of the one or more storage devices from memory using the buffer. The one or more processors may send a plurality of commands corresponding to the number of commands to the one or more storage devices based at least on the input data for each storage device and the identifier for each storage device.

Description

Technical Field

[0001] The present disclosure relates generally to systems and methods for providing an application with access to one or more storage devices, and more particularly to embedding multiple commands into one command to execute the multiple commands simultaneously on one or more storage devices. Background Art

[0002] An application on a host computer may access a storage device (e.g., a Small Computer System Interface (SCSI) device, a Parallel Advanced Technology Attachment (PATA) device, one or more Serial Advanced Technology Attachment (SATA) devices, one or more Serial Attached SCSI (SAS) devices, one or more Peripheral Component Interconnect Express (PCIe) devices, one or more Non-Volatile Memory Express (NVMe) devices) via a PT interface (e.g., a SCSI PT interface, a SATA PT interface, a SAS PT interface, a PCIe PT interface) using a pass-through (PT) command (e.g., a SCSI PT command, a SATA PT command, a SAS PT command, a PCIe PT command). When using the PT command, one management PT command (e.g., a Redundant Array of Independent Disks (RAID) management PT command) may be used to issue only one PT command to one storage device (e.g., a SCSI device, a PATA device, a STA device, a SAS device, a PCIe device, a NVMe device) at a time. Since the host driver allocates only a single host buffer for an application to issue a PT command, there may be a limitation that only one SCSI PT command is allowed at a time. Due to the limitation of only allowing one SCSIPT command to only one of the storage devices at a time, bulk operations on the storage devices need to be serialized. Since there is no parallelism, such serialized bulk operations may have a severe impact on performance. Summary of the invention

[0003] In one aspect, the present disclosure relates to a system comprising: one or more processors configured to: receive a frame comprising a number of commands, a number of storage devices, and a buffer map; in response to the frame, use the buffer map and read from a memory (1) input data corresponding to the number of one or more storage devices of each of the storage devices, and (2) an identifier of each of the one or more storage devices; and send a plurality of commands corresponding to the number of commands to the one or more storage devices based at least on the input data of each storage device and the identifier of each storage device.

[0004] In another aspect, the present disclosure relates to a device comprising: a memory; and a controller coupled to a plurality of storage devices, wherein the controller is configured to: receive a frame comprising a number of commands, a number of storage devices among the plurality of storage devices, and a buffer map; in response to the frame, use the buffer map and read from the memory (1) input data corresponding to the number of one or more storage devices of each of the storage devices, and (2) an identifier of each of the one or more storage devices; and send a plurality of commands corresponding to the number of commands to the one or more storage devices based at least on the input data of each storage device and the identifier of each storage device.

[0005] In other aspects, the present disclosure relates to a method comprising: receiving, by one or more processors, a frame comprising a number of commands, a number of storage devices, and a buffer mapping; in response to the frame, using the buffer mapping and reading from a memory (1) input data corresponding to each of the one or more storage devices of the number of storage devices, and (2) an identifier of each of the one or more storage devices; and sending, by the one or more processors, a plurality of commands corresponding to the number of commands to the one or more storage devices based at least on the input data of each storage device and the identifier of each storage device. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Various objects, aspects, features and advantages of the present disclosure will become more apparent and better understood by reference to the detailed description taken in conjunction with the accompanying drawings, in which like reference symbols identify corresponding elements throughout the text. In the drawings, like reference numerals generally indicate identical, functionally similar and / or structurally similar elements.

[0007] Figure 1A is a schematic block diagram of a system including an example storage device configuration according to an embodiment.

[0008] Figure 1B is a schematic block diagram of a computing system according to an embodiment.

[0009] Figure 2 is a schematic block diagram of a layout of a frame and a memory for executing a command in a storage device according to an embodiment.

[0010] Figure 3 is a schematic block diagram of a layout of frames and memories for executing multiple commands in one or more storage devices according to an embodiment.

[0011] Figure 4 is a schematic block diagram of a layout of buffer mapping and memory for executing multiple commands in one or more storage devices according to an embodiment.

[0012] Figure 5 is a flow chart showing a process for executing multiple commands in one or more storage devices according to an embodiment.

[0013] The details of various embodiments of the method and system are set forth in the accompanying drawings and the description below. DETAILED DESCRIPTION

[0014] The following disclosure provides many different embodiments or examples for implementing the different features of the subject matter provided. Specific examples of components and arrangements are described below to simplify the present disclosure. Of course, these are only examples and are not intended to be limiting. For example, a first feature communicating with a second feature or communicatively coupled to a second feature in the following description may include an embodiment in which the first feature directly communicates with the second feature or is directly coupled to the second feature, and may also include an embodiment in which an additional feature may be interposed between the first and second features so that the first feature indirectly communicates with the second feature or is indirectly coupled to the second feature. In addition, the present disclosure may repeat reference numbers and / or letters in various examples. This repetition is for the purpose of simplicity and clarity, and does not itself indicate the relationship between the various embodiments and / or configurations discussed.

[0015] Various embodiments disclosed herein are directed to a system comprising one or more processors configured to receive a frame, the frame comprising a number of commands (e.g., the number of commands), a number of storage devices (e.g., the number of storage devices), and a buffer map. In some embodiments, the frame may be a container for a network packet, a block of data for transmission, a digital data transmission unit, or any data transmission unit in a computer network. In response to the frame, the one or more processors may be configured to use the buffer map and read from a memory (1) input data for each of the one or more storage devices corresponding to the number of storage devices, and (2) an identifier for each of the one or more storage devices. In some embodiments, the memory may be a volatile memory (e.g., RAM), a non-volatile memory, a media and memory device, a semiconductor memory device (e.g., SDRAM and flash memory devices), a magnetic disk, a magneto-optical disk, or any device with a fixed storage device or a device for reading a removable storage medium, or any other system, device, or component that stores electronic data for use by a computer. In some embodiments, the identifier of each of the one or more storage devices may be a combination of a serial number, a device unique ID (DUID), a network address (e.g., an IP address, a MAC address), a vendor identifier, a product identifier, and / or a version identifier, or any value or character that can uniquely identify each storage device among the one or more storage devices. The one or more processors may be configured to send a plurality of commands corresponding to the number of commands to the one or more storage devices based at least on the input data of each storage device and the identifier of each storage device. In some embodiments, the storage device may be one or more volatile memory devices, one or more non-volatile memory devices, one or more magnetic disks, one or more magneto-optical disks, one or more hard disk drives (HDDs), one or more solid-state drives (SSDs), redundant arrays of independent disks (RAIDs), or any physical device suitable for storing computer-readable data. In some embodiments, the input data of the storage device may be any data used to execute commands in the storage device.

[0016] In some embodiments, the memory may include multiple buffers. The buffer map may identify the location and length of each of the multiple buffers. The multiple commands may be stored in a first buffer of the multiple buffers in an execution order. The one or more processors may be configured to read the multiple commands from the first buffer and send the multiple commands to the one or more storage devices in an execution order. In some embodiments, the buffer may be any temporary area where data is stored in the memory or storage device.

[0017] In some embodiments, identifiers of the one or more storage devices may be stored in a second buffer of the plurality of buffers. Input data of the one or more storage devices may be stored in a third buffer of the plurality of buffers. In some embodiments, the order in which the identifiers of the one or more storage devices are stored in the second buffer may be the same as the order in which the input data of the one or more storage devices are stored in the third buffer. The input and output buffers may be different for each command, and there may be multiple input buffers and multiple output buffers. The multiple input buffers and multiple output buffers may be arranged in the memory in any order.

[0018] In some embodiments, when sending the plurality of commands, the one or more processors may be configured to send a first command of the plurality of commands to the one or more storage devices. In response to sending the first command, the one or more processors may be configured to send a second command of the plurality of commands to the one or more storage devices without waiting for the completion of the execution of the first command on the one or more storage devices. In some embodiments, commands may be executed one after another on a particular storage device. However, a command may be executed on a particular storage device without waiting for the completion of the command from other storage devices (e.g., a storage device other than the particular storage device among the one or more storage devices). In some embodiments, a bit in a mailbox may be used to indicate that all commands may be executed in parallel, so that commands may be executed in parallel (even on the same storage device).

[0019] In some embodiments, one or more storage devices may form a redundant array of RAID devices. In some embodiments, a RAID device may include one or more storage devices (as defined above) or disks that form, define, or function as a RAID, or a controller configured to control a physical device or disk, or any device that can provide a RAID structure or function or service to a user. In some embodiments, multiple commands may be small computer system interface (SCSI) pass-through commands and may be sent as frames according to a message passing interface (MPI). In some embodiments, multiple commands may include one or more SCSI commands, one or more PATA commands, one or more SATA commands, one or more PCIe commands, one or more NVMe commands, one or more HDD commands, one or more SDD commands, or any command or instruction that can be sent to any physical device suitable for storing computer-readable data.

[0020] Various embodiments disclosed herein relate to a device that includes a memory and a controller coupled to a plurality of memory devices. The controller may be configured to receive a frame that includes a number of commands (e.g., the number of commands), a number of memory devices among a plurality of memory devices (e.g., the number of memory devices), and a buffer map. In response to the frame, the controller may be configured to use the buffer map and read from the memory (1) input data for each of one or more memory devices corresponding to the number of memory devices, and (2) an identifier for each of the one or more memory devices. The controller may be configured to send a plurality of commands corresponding to the number of commands to the one or more memory devices based at least on the input data for each memory device and the identifier for each memory device.

[0021] In some embodiments, the memory may include a plurality of buffers, and the buffer map may identify the location and length of each of the plurality of buffers. The plurality of commands may be stored in a first buffer of the plurality of buffers in an execution order. The controller may be configured to read the plurality of commands from the first buffer. The controller may be configured to send the plurality of commands to one or more storage devices in an execution order.

[0022] In some embodiments, identifiers of the one or more storage devices may be stored in a second buffer of the plurality of buffers. Input data of the one or more storage devices may be stored in a third buffer of the plurality of buffers.

[0023] In some embodiments, when sending multiple commands, the controller may be configured to send a first command of the multiple commands to the one or more storage devices. In response to sending the first command, the controller may be configured to send a second command of the multiple commands to the one or more storage devices without waiting for the completion of the execution of the first command on the one or more storage devices.

[0024] In some embodiments, one or more storage devices may form a RAID device. In some embodiments, the plurality of commands may be SCSI pass-through commands and may be sent as frames according to MPI.

[0025] Various embodiments disclosed herein are directed to a method. The method may include receiving, by one or more processors, a frame including a number of commands (e.g., the number of commands), a number of storage devices (e.g., the number of storage devices), and a buffer map. The method may include, in response to the frame, using, by the one or more processors, the buffer map and reading from a memory (1) input data for each of the one or more storage devices corresponding to the number of storage devices, and (2) an identifier for each of the one or more storage devices. The method may include sending, by the one or more processors, a plurality of commands corresponding to the number of commands to the one or more storage devices based at least on the input data for each storage device and the identifier for each storage device.

[0026] In some embodiments, the memory may include multiple buffers, and the buffer map may identify the location and length of each of the multiple buffers. Multiple commands may be stored in a first buffer of the multiple buffers in an execution order. One or more processors may read the multiple commands from the first buffer and send the multiple commands to the one or more storage devices in an execution order.

[0027] In some embodiments, identifiers of the one or more storage devices may be stored in a second buffer of the plurality of buffers. Input data of the one or more storage devices may be stored in a third buffer of the plurality of buffers.

[0028] In some embodiments, when sending multiple commands, the one or more processors may send a first command of the multiple commands to the one or more storage devices. In response to sending the first command, the one or more processors may send a second command of the multiple commands to the one or more storage devices without waiting for the completion of the execution of the first command on the one or more storage devices. In some embodiments, the one or more storage devices may form a RAID device.

[0029] Figure 1A and Figure 1B An example configuration of a storage system (eg, a RAID system) is presented. Figure 1A 1 is a schematic block diagram of a system 100 including a host 110, a storage controller 130 (e.g., a RAID controller), and a plurality of physical (storage) devices (PD-1, ..., PD-N) 150-1, ..., 150-N. For example, the storage controller 130 may control or support thousands of PDs 150. Each of the host 110, the storage controller 130, and the PDs 150 may have a configuration similar to Figure 1B Although each of the host 110, the storage controller 130, and the PD 150 does not need to have a configuration of the computing system in Figure 1B In some embodiments, each physical device may be one or more volatile memory devices, one or more non-volatile memory devices, one or more magnetic disks, one or more magneto-optical disks, one or more HDDs, one or more SSDs, RAID, or any physical device suitable for storing computer-readable data.

[0030] The host 110 may include a memory 118, one or more applications 111, and a storage controller driver (or host driver) 116. Each of the one or more applications 111 and the host driver 116 may be implemented in hardware, software, firmware, or any combination thereof. The one or more applications 122 may send commands to physical devices (PDs) 150-1, ..., 150-N via the host driver 116 and the storage controller 130 using communication between processes running on one or more processors (e.g., CPUs). In some embodiments, the one or more applications 122 may send commands to the PDs according to a message passing interface (MPI) standard (e.g., MPI 3.0). In some embodiments, MPI may include any type and form of interface for communicating messages between computer processes, such as, for example, communicating messages among processes of a host application, a storage controller, and / or a PD according to any MPI standard and any version thereof.

[0031] In some embodiments, the command may be a SCSI pass-through (PT) command, and the application 111 may send a SCSI PT command (e.g., a RAID management PT command) to the PD 150 on the storage controller 130 through the MPI management request queue. The SCSI PT command may be provided as a SCSI pass-through interface so that the application 111 can access the SCSI device using the SCSI PT command. During the host driver negotiation, the host driver 116 may allocate one or more host buffers 119 (e.g., a single buffer of 1MB in size) in the memory 118 to transfer input, data, and status between the application 111 and the storage controller 130. In some embodiments, the memory 118 may be a memory with direct memory access (DMA) capability so that DMA is used to transfer data between the memory 118 and the local memory of the storage controller 130 using the host buffers 119.

[0032] In some embodiments, the storage controller 130 may receive a command (e.g., a RAID management PT command) sent from the application 111 as a request frame 112 (e.g., an MPI 3.0 request frame). The request frame 112 for sending a command (e.g., a PT command) may have a command scatter list (SGL) and a response SGL. A context 113 (e.g., a PT context) carrying information of the command may be stored in the command SGL and may be transferred to a local buffer of the storage controller 130 via the DMA 120. For example, the local buffer may be a local message index (LMID) context for storing LMID operation specific data. If the PT command is of type DCMD (data command or direct command), then the context 113 may store information of the DCMD command. The DCMD command may be a contract between an application (e.g., application 111) and a storage controller (e.g., controller 130) to (1) send management commands (along with parameters and / or input data) from the application to the storage controller, and (2) receive status and data in return. For example, the application 111 may use the DCMD command to download a disk image from the PD 150. DCMD input or output data may be transferred via one or more buffers demarcated from host buffer 119. Data transfers to and from these buffers may use DMA 120. Figure 2 Describes the details of the request frame and the layout of the buffer.

[0033] In some embodiments, in response to receiving the request frame 112 carrying the command, the storage controller 130 may send the command to the PD so that the PD may execute the command and return the execution result to the storage controller 130. In response to receiving the execution result, the storage controller 130 may send the execution result to the application 111 as a response frame 114 (e.g., an MPI 3.0 response frame). The response frame 114 may have a layout or format similar to the request frame 112. For example, a context 115 (e.g., a PT context) carrying execution information may be stored in a response SGL of the response frame and may be transferred from a local buffer of the storage controller 130 to one or more buffers demarcated from the host buffer 119 via the DMA 120.

[0034] Figure 1B1 is a schematic block diagram of a computing system according to an embodiment. The illustrated example computing system 1000 includes one or more processors 1010 communicating with memory 1060 via a communication system 1040 (e.g., a bus), at least one network interface controller 1030 having a network interface port for connecting to a network (not shown), and other components, such as input / output ("I / O") components 1050. Typically, the processor 1010 will execute instructions (or computer programs) received from the memory. The illustrated processor 1010 incorporates or is directly connected to a cache memory 1020. In some examples, instructions are read from the memory 106 into the cache memory 1020 and executed by the processor 1010 from the cache memory 1020. The computing system 1000 may not necessarily contain Figure 1B All of these components shown in the Figure 1B Other components not shown.

[0035] In more detail, processor 1010 may be any logic circuitry that processes instructions, for example, instructions extracted from memory 1060 or cache 1020. In many embodiments, processor 1010 is a microprocessor unit or a special purpose processor. Computing device 1050 may be based on any processor or processor group capable of operating as described herein. Processor 1010 may be a single core or multi-core processor. Processor 1010 may be a plurality of different processors.

[0036] The memory 1060 may be any device suitable for storing computer-readable data. The memory 1060 may be a device with fixed storage or a device for reading removable storage media. Examples include all forms of volatile memory (e.g., RAM), nonvolatile memory, media and memory devices, semiconductor memory devices (e.g., EPROM, EEPROM, SDRAM, and flash memory devices), magnetic, magneto-optical disks, and optical disks (e.g., CD ROM, DVD-ROM, or Computing system 1000 may have any number of memory devices 1060 .

[0037] Cache memory 1020 is typically a form of computer memory placed in close proximity to processor 1010 for fast read times. In some implementations, cache memory 1020 is part of processor 1010, or is on the same chip as processor 1010. In some implementations, there are multiple levels of cache 1020, for example, L2 and L3 cache layers.

[0038] The network interface controller 1030 manages data exchange via a network interface (sometimes referred to as a network interface port). The network interface controller 1030 handles the physical and data link layers of the OSI model for network communication. In some embodiments, some of the tasks of the network interface controller are handled by one or more of the processors 1010. In some embodiments, the network interface controller 1030 is part of the processor 1010. In some embodiments, the computing system 1000 has multiple network interfaces controlled by a single controller 1030. In some embodiments, the computing system 105 has multiple network interface controllers 1030. In some embodiments, each network interface is a connection point for a physical network link (e.g., a Category 5 Ethernet link). In some embodiments, the network interface controller 1030 supports wireless network connections, and the interface port is a wireless (e.g., radio) receiver or transmitter (e.g., for any of the IEEE 802.11 protocols, near field communication "NFC", Bluetooth, ANT, or any other wireless protocols). In some embodiments, the network interface controller 1030 implements one or more network protocols (e.g., Ethernet). Typically, computing device 1050 exchanges data with other computing devices via a physical or wireless link through a network interface. The network interface may be linked directly to another device or linked to another device via an intermediate device, such as a network device that connects computing device 1000 to a data network (such as the Internet), such as a hub, bridge, switch, or router.

[0039] The computing system 1000 may include, or provide an interface for, one or more input or output ("I / O") devices. Input devices include, but are not limited to, keyboards, microphones, touch screens, foot switches, sensors, MIDI devices, and pointer devices (such as a mouse or trackball). Output devices include, but are not limited to, video displays, speakers, refreshable Braille terminals, lights, MIDI devices, and 2-D or 3-D printers.

[0040] Other components may include I / O interfaces, external serial device ports, and any additional co-processors. For example, computing system 1000 may include interfaces (e.g., Universal Serial Bus (USB) interfaces) for connecting input devices, output devices, or additional memory devices (e.g., portable flash drives or external media drives). In some embodiments, computing device 1000 includes additional devices, such as co-processors, for example, mathematical co-processors that can assist processor 1010 with high precision or complex calculations.

[0041] Component 1090 may be configured to connect to external media, display 1070, input device 1080, or any other component or combination thereof in computing system 1000. Display 1070 may be a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, a cathode ray tube (CRT), a projector, a printer, or other now known or later developed display device for outputting determined information. Display 1070 may serve as an interface for a user to view the operation of processor 1010, or specifically as an interface with software stored in memory 1060.

[0042] The input device 1080 may be configured to allow a user to interact with any of the components of the computing system 1000. The input device 1080 may be a plurality of pads, a keyboard, a cursor control device (such as a mouse), or a joystick. In addition, the input device 1080 may be a remote control, a touch screen display (which may be a combination of the display 1070 and the input device 1080), or any other device operable to interact with the computing system 1000, such as any device operable to act as an interface between a user and the computing system 1000.

[0043] Figure 2 2 is a schematic block diagram of a layout 200 of a frame (eg, a request frame 210 carrying a DCMD command) and a memory (eg, a host memory 240) for executing a command in a storage device according to an embodiment. Figure 2 , the DCMD command may be embedded, included, identified, specified, or contained within a context 220 (e.g., a RAID management PT context) of a request frame (e.g., an MPI 3.0 request frame). In one embodiment, the context 220 for the DCMD command may include fields for a management opcode 222 (e.g., a management PT opcode), a DCMD opcode 224, a mailbox 226, and / or request or response information 228. The management opcode field 222 may contain a management opcode type (e.g., a "DCMD" type in the current context). The DCMD opcode field 224 may contain a DCMD opcode, which is a value used to identify the DCMD command. The mailbox field 226 may be a buffer for passing data together with the DCMD opcode. In some embodiments, the mailbox field 226 may be a buffer of 32 bytes in size that is sent to the storage controller (e.g., the storage controller 130) in the PT context 220 along with the DCMD command. Since the size of the mailbox field is limited, the mailbox may be used only to pass information bits and bytes, such as PD identifiers, general indicators, etc. The general indicator may be a flag related to the DCMD, for example, a flag for discarding the cache before deleting the LD (if the DCMD is to delete the LD).

[0044] The request or response information field 228 may contain DCMD information (e.g., command SGL and / or response SGL) and buffer information. In some embodiments, the buffer information may include a number of sub-buffers of a host buffer in a host memory (e.g., sub-buffers 250-1, ..., 250-n of a host buffer 242 in memory 240), an offset and length of each sub-buffer (or any location information of the sub-buffer), and a buffer number or identifier of each sub-buffer. In some embodiments, the buffer information may be represented by an SGL and contain information pointing to or linking to (or locating) n sub-buffers 250-1, ..., 250-n in memory 240 (e.g., offset and length of each sub-buffer). In some embodiments, the sub-buffers 250-1, ..., 250-n may be from a single host buffer (e.g., Figure 1A 119 in the host buffer 119). Subbuffers 250-1, ..., 250-n may be DCMD specific subbuffers with predefined offsets and lengths in the host buffer. The subbuffers may be continuous in the memory, but may be discontinuous in some embodiments. Each subbuffer may be identified by a buffer number. In some embodiments, the subbuffer may be of variable length even for the same DCMD command. In some embodiments, the total number of subbuffers and the number of buffers may be predetermined or predefined for the management PT command. That is, since only one management PT command (e.g., RAID management PT command) is supported in one DCMD command, the total number of subbuffers and the number of buffers may be defined as fixed for each DCMD command. Using subbuffers, the host buffer may be used for bulk data transfer of DCMD together with DCMD opcodes and parameters, which may be variable counts and sizes and are clearly defined and understood by both the application and the storage controller.

[0045] In one aspect, an application (e.g., a program) on a host computer can access a storage device using a PT command via a PT interface. When using a PT command, one management PT command (e.g., a RAID management PT command) can be used to issue only one PT command to one storage device (e.g., a SCSI device) at a time. Since the host driver only allocates a single host buffer for the application to issue a PT command, there may be a limitation that only one SCSIPT command is allowed at a time. Alternatively, multiple SCSIPT commands can be used through an out-of-band (OOB) interface. However, the number of SCSIPT commands may also be limited by the number of buffers (usually 2) available for the OOB interface. Since only one SCSIPT command is allowed to access only one of the storage devices at a time, batch operations on the storage device need to be serialized. Since there is no parallelism, such serialized batch operations may have a serious impact on performance. For example, even if the same SCSIPT command must be issued to multiple PDs, the issuance to multiple PDs cannot be achieved in parallel. Even for multiple PDs (e.g., n PDs) of the same model and comparable speed, issuing commands to n PDs may take n times the duration of completing the command on one PD.

[0046] Furthermore, in a series of operations, a single slow command on a PD can delay the entire process. For example, assuming a system in which there are 2 PDs, the first command (Command 1) may take 1 second on both PDs, and the second command (Command 2) may take 5 seconds on PD1 and 2 seconds on PD2. When executed sequentially, Command 1 will take 2 (=1+1) seconds to complete on both devices (e.g., drives), while Command 2 will take 7 (=5+2) seconds to complete on both devices, resulting in a total time of 9 seconds. With thousands of PDs, this delay can add up to hours.

[0047] Furthermore, even if cumulative bidirectional data transfers are performed within the host buffer size, multiple commands cannot be issued in parallel. The SCSI protocol does not support issuing commands to multiple endpoints simultaneously. Therefore, storage solutions also follow the same approach (e.g., serial bulk operations) and do not achieve parallelism. Furthermore, in most cases, the host buffer is suboptimally utilized because the data transfer involved in executing a single non-I / O SCSI command on a single PD is not large compared to the host buffer size. Therefore, even if the host buffer size is increased, space will be wasted when executing most SCSIPT commands.

[0048] To address these issues, according to certain aspects, embodiments in the present disclosure relate to a technique for embedding, combining, or multiplexing multiple storage commands (e.g., SCSIPT commands) in one command (e.g., one DCMD command or one management PT command) and issuing multiple storage commands to multiple PDs simultaneously. For example, 256 SCSIPT commands may be embedded, contained, identified, specified, or included in one DCMD command and may be issued or executed simultaneously. In some embodiments, a system for executing multiple commands in one or more storage devices may allow both fixed and (optionally) dynamic amounts of bidirectional data transfer for each PD while sending the command status (or execution status) of each PD back to the application. Compared to techniques in which only one command (e.g., a DCMD command) is supported and the total number of commands and buffers (usually 1 for request data and 1 for response data) is clearly defined, in some embodiments, multiple commands may be issued together so that each command may use a different number of buffers depending on the type of command. Thus, the total number of buffers and sub-buffers (or segments) may vary.

[0049] In some embodiments, a command (e.g., a new type of DCMD command) for multiplexing multiple PT commands (e.g., SCSI PT commands) may be embedded, included, identified, specified, or contained within a context (e.g., a RAID management PT context) of a request frame (e.g., an MPI 3.0 request frame). The context of the new DCMD command may include the following fields: (1) a management opcode (e.g., a management PT opcode), (2) a DCMD opcode indicating a new DCMD command (e.g., a multiplexed DCMD), (3) a mailbox, and / or (4) a buffer map. The management opcode field may contain a management opcode type (e.g., a "DCMD" type). The DCMD opcode field may contain a new (or newly defined) DCMD opcode (e.g., a value) indicating the type of the multiplexed DCMD. The multiplexed DCMD command may support the execution of multiple PT commands on one or more PDs simultaneously.

[0050] The mailbox field may be a buffer (e.g., a 32-byte buffer represented by mailbox[0...31]) used to pass data along with an opcode (e.g., a management opcode or a DCMD opcode). The mailbox field may be used primarily to pass general information, such as information related to checking the status of a PD prior to an operation. In some embodiments, the mailbox field may contain (1) the number of commands, such as SCSIPT commands (which are embedded, contained, identified, specified, or contained in a multiplexed DCMD command and may be issued or executed simultaneously or concurrently) and / or (2) the number of PDs to which each command may be issued. In some embodiments, a byte or half byte in the mailbox may be used to indicate the number of SCSIPT commands in a frame. If conventional PT commands are used, the number of SCSIPT commands that may be executed serially may be limited because, in most cases, a SCSIPT command is issued by an application only after the status of a previous command has been verified as successful. In contrast, in some embodiments, the storage controller may continue to issue the next command on behalf of the application unless a successful status check is required for the previous command. In some embodiments, a byte in the mailbox (different from the byte or nibble used for the number of SCSI PT commands) may be used to specify the number of PDs to which each SCSI PT command is to be issued.

[0051] The buffer map field (also referred to as buffer map) may contain information (e.g., information used to access the buffers) about a plurality of buffers in the memory (represented by buffer [0], buffer [1], ..., buffer [n] of the host buffers in the host memory, where n is an integer greater than or equal to 1). Details of the plurality of buffers and the buffer map field (or buffer map) will be described in the following paragraphs.

[0052] In some embodiments, multiple buffers of the host buffer may be allocated, assigned, or drawn from the host memory. The multiple buffers may include one or more bidirectional buffers that contain both input data and output data for the PT command. Each buffer may be divided into multiple segments. Each segment may contain common data for all PDs or PD specific data. The length of each segment (segment length) may be fixed or variable. In some embodiments, the multiple buffers may include a segment length buffer to define or specify a fixed segment length or a variable segment length.

[0053] In some embodiments, the plurality of buffers may include a PT command buffer (e.g., a SCSIPT command buffer). In some embodiments, m PT request frames (e.g., MPI 3.0 SCSI PT request frames) may be embedded, included, identified, specified, or contained in one buffer, e.g., a PT command buffer, where m is an integer greater than or equal to 1. Using the PT command buffer, a system for executing multiple PT commands (e.g., SCSI PT commands) may operate like a multiplexer through which multiple SCSI PT commands may be sent simultaneously from an application to a storage controller. For example, each of the SCSI PT frames may be 128 bytes in length. In the PT command buffer, the SCSI PT frames may be packaged one after another in the same order in which the application wants the storage controller to execute the commands. Since the commands contained in the SCSI PT frames are to be executed on each of one or more devices (PDs), any device specific fields (e.g., device disposition fields) of the SCSI PT frames representing a particular single device (PD) may be marked as invalid.

[0054] In some embodiments, a PT command may have a command number defined by its position in a PT command buffer. For example, a PT command in a first position in a PT command buffer may be command number 1, a PT command in a second position in a PT command buffer may be command number 2, and so on. In some embodiments, a PT command may have up to 1 input buffer and 1 output buffer. A PT command may be linked to its corresponding input buffer and output buffer by linking the command number of the PT command to the buffer numbers of the corresponding input buffer and output buffer. In some embodiments, a buffer map may link the command number of a PT command to the buffer numbers of the input buffer and output buffer of the PT command (or have information linking the command number of a PT command to the buffer numbers of the input buffer and output buffer of the PT command).

[0055] In some embodiments, the plurality of buffers may include a device list buffer that lists all PDs to which SCSIPT commands are issued (e.g., lists PD identifiers of all PDs). The device list buffer may define or specify (1) the number of segments (units that carry data for each PD in the device list) in the buffer that are not common to all PDs, and (2) the order in which the segments of each PD are stored in the buffer. For example, if there are n devices in the device list, then other buffers (sub-buffers with fixed lengths) that carry information for PD-1, PD-2, ..., and PD-n may use the length of the data for each of the PDs (total buffer length / n). In some embodiments, the device list buffer may be used to implement multiplexed DCMD even if there is only one PD.

[0056] In some embodiments, the plurality of buffers may include one or more segment length buffers to define or specify variable length segments in the buffer (for corresponding PDs). The segment length buffer may contain or specify the length of each segment in the buffer (for corresponding PDs) in the same order as the PDs listed in the device list buffer. All elements in the segment length buffer may be the same length and represent corresponding PDs in the same order as the PDs listed in the device list buffer. Like other buffers, the segment length buffer may have a buffer number.

[0057] The input data may be common to all PDs or PD specific. In some embodiments, multiple buffers may include an input buffer to contain PD specific input data. For example, if n bytes of data must be sent for each PD, and there are m PDs to which the command is issued, then the input buffer may have m segments of n bytes each. The m segments may be arranged or positioned in the same order as the PDs listed in the device list buffer. For example, input data for the first device listed in the device list buffer may be contained in the first segment of the input buffer, input data for the second device listed in the device list buffer may be contained in the second segment of the input buffer, and so on. If no input data is used for a command (e.g., a DCMD command), then this buffer may be omitted or optional. A segment length buffer may be used to specify or define variable length segments per PD of the input buffer.

[0058] In some embodiments, the multiple buffers may include an output buffer for returning output data for each PD. For example, if n bytes of data must be returned for each PD, and there are m PDs to which commands are issued, then the output buffer may have m segments of n bytes each. The m segments may be arranged or positioned in the same order as the PDs listed in the device list buffer. If output data from the PDs for a command (e.g., a DCMD command) is not expected, then this buffer may be omitted or optional. A segment length buffer may be used to specify or define variable length segments per PD of the output buffer.

[0059] In some embodiments, the multiple buffers may include one or more buffers that can transfer both input and output data. In other words, the same buffer may be marked or used for both input and output, and such buffers may be indicated as overlapping buffers or bidirectional buffers. After the storage controller issues commands to multiple PDs, data returned from multiple PDs may be received at different times. Therefore, the storage controller may complete the buffer DMA to the local memory before starting an operation on the first PD (e.g., issuing the next command). For example, as and when data is received from individual PDs, the storage controller may use individual DMA to write data to the output buffer instead of waiting for the output of all PDs. In some embodiments, each buffer may have a data direction - input (0), output (1), or bidirectional (2). In other words, by specifying the data direction of each buffer (e.g., in the buffer map), each buffer may be indicated as an input buffer, an output buffer, or a bidirectional buffer.

[0060] In some embodiments, the plurality of buffers may include a failed command number buffer having an array of bytes or nibbles sorted in device list order (e.g., the order of the PDs listed in the device list buffer), each byte or nibble indicating a command number that has failed for the corresponding PD. One command from a series of commands may have failed for the PD, and the storage controller stops issuing unprocessed commands (if any) to the PD. In this case, to indicate to the application which command failed, the storage controller may set the command number in an element corresponding to the PD in the failed command number buffer. For example, if the second command fails for the PD listed in the third element of the device list buffer, the storage controller may set the corresponding element for the PD (i.e., the third element of the failed command number buffer) to the corresponding command number (i.e., the command number corresponding to the second command). If all commands succeed, all elements of the failed command number buffer may be set to a value of 0. If only one SCSIPT command is embedded, contained, identified, specified, or contained, the failed command number buffer may be omitted or optional.

[0061] In some embodiments, the plurality of buffers may include a status buffer to indicate the status (e.g., execution status) of a command. The status of a command may indicate a failure reason or code for a failed command. For example, the status buffer may have an array of command execution status for each PD sorted in device list order. In some embodiments, the status of the last command executed on each PD may be stored in the status buffer.

[0062] In some embodiments, the buffer map field may contain the total number of buffers used in the current DCMD command (e.g., (n+1) buffers with buffer numbers 0, 1, 2, ..., n), and buffer access information for accessing each of the multiple buffers. For each buffer, the buffer access information may include the buffer number, offset, length, segment length, number of elements, and / or data direction of the buffer. In some embodiments, a segment length of 0 may indicate a common buffer for all PDs (e.g., a buffer with PD firmware images for all PDs listed in the device list buffer). The common buffer may be common to all PDs and have no sub-buffers. For example, the PT command buffer and the device list buffer may not be a common buffer because both have (fixed-length) sub-buffers.

[0063] In some embodiments, buffer number 0 may be reserved for the PT command buffer, and buffer number 1 may be reserved for the device list buffer. For example, the buffer map may include buffer access information for a PT command buffer (e.g., a list of MPI 3.0 SCSI PT request frames of 128 bytes each) having buffer number 0 (buffer [0]). The buffer map may include buffer access information for a device list buffer (e.g., an array of PD identifiers) having buffer number 1 (buffer [1]). The buffer map may include buffer access information for a failed command number buffer, such as buffer number (n-1). The buffer map may include buffer access information for a status buffer, such as buffer number n.

[0064] In some embodiments, the buffer map may contain mapping information. For example, the buffer map may specify a mapping between a command number (of a command) and a buffer number of an input buffer (corresponding to the command). The buffer map may specify a mapping between a command number (of a command) and a buffer number of an output buffer (corresponding to the command).

[0065] In some embodiments, the buffer map may contain information of one or more segment length buffers, such as the buffer number and the use of each segment length buffer. In some embodiments, if a segment length buffer is used for a particular buffer, the buffer access information for the particular buffer may include (1) a bit indicating the use of the segment length buffer, and (2) the buffer number of the segment length buffer to be used.

[0066] The embodiments of the present disclosure have at least the following advantages and benefits. First, the embodiments of the present disclosure may provide a useful technique for issuing multiple commands (e.g., SCSIPT commands) to multiple PDs simultaneously in a multiplexed manner. For example, a first command is issued to multiple PDs, then a second command is issued to multiple PDs, and so on. Therefore, the time taken to complete a series of commands to multiple PDs with comparable performance may be equal to the time taken to complete a series of commands on one such PD. The PDs may not necessarily be of the same model. In some embodiments, multiple SCSIPT commands may be issued simultaneously as long as the cumulative data transfer in any direction is less than the size of a host buffer (e.g., a host buffer comprising multiple buffers).

[0067] Second, the embodiments in the present disclosure may provide useful techniques for executing commands in parallel on multiple PDs. For example, the same command may be executed in parallel on multiple PDs, or different commands may be executed in parallel on multiple PDs. As a result, the command may take the time taken by the slowest PD to complete. Assuming a system in which there are 2 PDs, the first command (command 1) may take 1 second on both PDs, and the second command (command 2) may take 5 seconds on PD1 and 2 seconds on PD2. When executed sequentially, command 1 will take 2 (=1+1) seconds to complete on two devices (e.g., drives), while command 2 will take 7 (=5+2) seconds to complete on both devices, resulting in a total time of 9 seconds. In contrast, if commands are executed in parallel on two PDs, command 1 may take 1 second to complete, and command 2 may take 5 seconds (the time taken to complete on the slowest PD) to complete, so when the commands are executed sequentially, the process may take 6 seconds instead of 9 seconds to complete. In other words, as the number of commands increases, the time to complete the command may not increase linearly.

[0068] Third, embodiments of the present disclosure may provide useful techniques for executing commands without waiting for the completion of previous commands. In some embodiments, the storage controller may not need to wait for a command to complete on all PDs. When a previous command completes, the next command may be issued immediately on the PD without waiting for any other commands to complete. In some embodiments, if multiple commands can be issued to the same PD, then multiple commands can be executed in parallel on the PD. For example, a bit in a mailbox may be used to indicate that all commands in the SCSIPT command buffer may be issued in parallel to each PD in the device list buffer, so that multiple commands may be executed in parallel on each PD.

[0069] Fourth, embodiments of the present disclosure may provide useful techniques for optimally utilizing the host buffer. In some embodiments, the host buffer may be optimally utilized even if the host buffer size is increased because more SCSIPT commands and PDs may be accommodated in the host buffer. For example, increasing the host buffer size from 1MB to 4MB may significantly improve the performance of sequential command execution per PD (e.g., in the case of executing commands such as PD firmware download, query data extraction, etc.)

[0070] Figure 3 FIG. 3 is a schematic block diagram of a layout 300 of a frame (eg, request frame 310) and a memory (eg, host memory 340) for executing multiple commands in one or more storage devices according to an embodiment. Figure 3 , a command (e.g., a new type of DCMD command) for multiplexing multiple PT commands (e.g., SCSI PT commands) may be embedded, included, identified, specified, or contained in a context 320 (e.g., a RAID management PT context) of a request frame 310 (e.g., an MPI 3.0 request frame). The context 320 of the new DCMD command may include the following fields: (1) a management opcode 322 (e.g., a management PT opcode), (2) a DCMD opcode 324 indicating a new DCMD command (e.g., a multiplexed DCMD), (3) a mailbox 326, and / or (4) a buffer map 330. The management opcode field 322 may contain a management opcode type (e.g., a "DCMD" type). The DCMD opcode field 324 may contain a new (or newly defined) DCMD opcode (e.g., a value) indicating the type of the multiplexed DCMD. The multiplexed DCMD command may support the execution of multiple PT commands on one or more PDs simultaneously.

[0071] The mailbox field 326 may be a buffer (e.g., a 32-byte buffer represented by mailbox [0 ... 31]) that passes data along with an opcode (e.g., a management opcode 322 and / or a DCMD opcode 324). The mailbox field 326 may be used primarily to pass general information, such as information related to checking the status of a PD prior to an operation. In some embodiments, the mailbox field 326 may contain (1) a number 327 of commands, such as SCSIPT commands (which are embedded, contained, identified, specified, or contained in a multiplexed DCMD command and may be issued or executed simultaneously or concurrently) and / or (2) a number 328 of PDs to which each SCSIPT command may be issued. A byte or half byte in the mailbox may be used to indicate the number of SCSIPT commands embedded, contained, identified, specified, or contained. A byte in the mailbox (different from the byte or half byte used for the number of SCSI PT commands) may be used to specify the number of PDs to which each SCSIPT command is to be issued.

[0072] The buffer map field 330 may contain information (e.g., information for accessing buffers) about a plurality of buffers in the memory (represented by buffer [0] 350-0, buffer [1] 350-1, ..., buffer [n] 350-n of the host buffer 342 in the host memory 340, where n is an integer greater than or equal to 1). Figure 4 Describes details of multiple buffers and the buffer map field (or buffer map).

[0073] Figure 4 4 is a schematic block diagram of a layout 400 of a buffer map 430 and a host memory 440 for executing multiple commands in one or more storage devices according to an embodiment. Figure 4 , multiple buffers 450 (buffer [0] 450-0, buffer [1] 450-1, ..., buffer [n] 450-n) of the host buffer 442 can be allocated from the host memory 440. The (n+1) buffers can have corresponding buffer numbers (e.g., 0, 1, 2, ..., n). Each buffer can be divided into multiple segments (or sub-buffers). For example, buffer [0] has m segments, where m is an integer greater than or equal to 1. In some embodiments, the number m corresponds to the number of commands embedded, contained, identified, specified, or contained in the PT command buffer. Each of buffers [1] to buffer [n] has k segments, where k is an integer greater than or equal to 1. In some embodiments, the number k corresponds to the number of PDs whose identifiers are embedded, contained, identified, specified, or contained in the device list buffer. Each segment can contain common data for all PDs or PD-specific data. The length of each segment (segment length) can be fixed or variable. The plurality of buffers may include a segment length buffer to define or specify a fixed segment length or a variable segment length.

[0074] refer to Figure 4, the plurality of buffers may include a PT command buffer (e.g., buffer [0] 450-0). In some embodiments, m PT request frames may be embedded, included, identified, specified, or contained in the PT command buffer, where m is an integer greater than or equal to 1. Using the PT command buffer 450-0, a system for executing multiple PT commands (e.g., SCSI PT commands) may function like a multiplexer through which multiple SCSI PT commands may be sent simultaneously from an application to a storage controller. For example, each of the SCSI PT frames may be 128 bytes in length. In the PT command buffer 450-0, the SCSI PT frames may be packaged one after another in the same order in which the application wants the storage controller to execute the commands. The PT commands may have command numbers defined by their positions in the PT command buffer 450-0. For example, the PT command in the first position in the PT command buffer 450-0 may be command number 1, the PT command in the second position in the PT command buffer may be command number 2, and so on. The PT command may have up to 1 input buffer and 1 output buffer. The PT command may be linked to its corresponding input buffer and output buffer by linking the command number of the PT command to the buffer numbers of the corresponding input buffer and output buffer. The buffer map 430 may link the command number of the PT command to the buffer numbers of the input buffer and output buffer of the PT command (or have information linking the command number of the PT command to the buffer numbers of the input buffer and output buffer of the PT command).

[0075] The plurality of buffers may include a device list buffer (e.g., buffer [1] 450-1) that lists all PDs to which SCSIPT commands are issued (e.g., lists PD identifiers or references for all PDs). The device list buffer may define or specify: (1) the number of segments (units that carry data for each PD in the device list) in the buffer that are not common to all PDs; and (2) the order in which the segments for each PD are stored in the buffer. For example, device list buffer 450-1 specifies: (1) m segments; and (2) the order of PD ID 0, PD ID 1, ..., PD ID m.

[0076] The plurality of buffers may include one or more segment length buffers to define or specify variable length segments in the buffer (for corresponding PDs). The segment length buffer may contain or specify the length of each segment in the buffer (for corresponding PDs) in the same order as the PDs listed in the device list buffer 450-1. All elements in the segment length buffer may be the same length and represent corresponding PDs in the same order as the PDs listed in the device list buffer 450-1. Like other buffers, the segment length buffer may have a buffer number.

[0077] The plurality of buffers may include an input buffer to contain PD-specific input data. In some embodiments, the input data may be data to be stored, accessed, or managed via the PD or any data that may be input to the PD. For example, if n bytes of data must be sent for each PD, and there are m PDs to which the command is issued, the input buffer may have m segments of n bytes each. The m segments may be arranged or positioned in the same order as the PDs listed in the device list buffer 450-1. For example, the input data for the first device listed in the device list buffer (e.g., PD ID 0) may be contained in the first segment (e.g., segment 0) of the input buffer, the input data for the second device listed in the device list buffer (e.g., PD ID 1) may be contained in the second segment (e.g., segment 1) of the input buffer, and so on. If no input data is used for a command (e.g., a DCMD command), the input buffer may be omitted or optional. A segment length buffer may be used to specify or define variable length segments per PD of the input buffer.

[0078] Similar to the input buffer, the multiple buffers may include an output buffer for returning output data for each PD. For example, if n bytes of data must be returned for each PD, and there are m PDs to which the command is issued, then the output buffer may have m segments of n bytes each. The m segments may be arranged or positioned in the same order as the PDs listed in the device list buffer 450-1. If output data from the PDs for a command (e.g., a DCMD command) is not expected, then the output buffer may be omitted or optional. The segment length buffer may be used to specify or define variable length segments per PD of the output buffer.

[0079] In some embodiments, the multiple buffers may include one or more buffers that can transfer both input and output data. In other words, the same buffer may be marked or used for both input and output, and such buffers may be indicated as overlapping buffers or bidirectional buffers. After the storage controller issues commands to multiple PDs, data returned from the multiple PDs may be received at different times. Therefore, the storage controller may complete the buffer DMA to the local memory before starting an operation on the first PD (e.g., issuing the next command). For example, as and when data is received from individual PDs, the storage controller may use individual DMA to write the data to the output buffer instead of waiting for the output of all PDs. In some embodiments, each buffer may have a data direction that is one of input (0), output (1), or bidirectional (2). In other words, by specifying the data direction of each buffer (e.g., in the buffer map 430), each buffer may be indicated as an input buffer, an output buffer, or a bidirectional buffer.

[0080] refer to Figure 4 , the plurality of buffers may include a failed command number buffer (e.g., buffer [n-1] 450-(n-1)) having an array of bytes or nibbles sorted in device list order (e.g., the order of the PDs listed in device list buffer 450-1), each byte or nibble indicating a command number that has failed for the corresponding PD. One command from a series of commands may have failed for the PD, and the storage controller stops issuing unprocessed commands (if any) to the PD. In this case, to indicate to the application which command has failed, the storage controller may set the command number in an element corresponding to the PD in failed command number buffer 450-(n-1). For example, if the second command (e.g., command 1) fails for the PD listed in the third element (e.g., PD ID 2) of the device list buffer, the storage controller may set the corresponding element of the PD (i.e., segment 2 of failed command number buffer 450-(n-1)) to the corresponding command number (i.e., 1). If all commands are successful, all elements of failed command number buffer 450-(n-1) may be set to a value of 0. If only one SCSIPT command is embedded, contained, identified, specified, or included, then failed command number buffer 450-(n-1) may be omitted or optional.

[0081] In some embodiments, the plurality of buffers may include a status buffer (e.g., buffer [n] 450-n) to indicate the status of a command (e.g., execution status). The status of a command may indicate a failure reason or code for a failed command. For example, the status buffer 450-n may have an array of command execution status for each PD sorted in device list order. In some embodiments, the status of the last command executed on each PD may be stored in the status buffer 450-n.

[0082] refer to Figure 4 , the buffer map 430 may contain the total number of buffers 431 used in the current DCMD command (e.g., (n+1) buffers), and buffer access information for accessing each of the multiple buffers (e.g., buffer access information 432, 433, 434-2, ..., 434-n, 438, 439). For each buffer, the buffer access information may include a buffer number (e.g., 0, 1, ..., n corresponding to buffer [0], buffer [1], ..., buffer [n]), an offset (in the host buffer 442), a length, a segment length, a number of elements (e.g., the number of segments), and / or a data direction of the buffer. In some embodiments, a segment length of 0 may indicate a common buffer for all PDs (e.g., a PT command buffer, a device list buffer).

[0083] In some embodiments, buffer number 0 may be reserved for a PT command buffer (e.g., buffer [0] 450-0), and buffer number 1 may be reserved for a device list buffer (buffer [1] 450-1). For example, the buffer map 430 may include buffer access information 432 for a PT command buffer 450-0 (e.g., a list of MPI 3.0 SCSI PT request frames of 128 bytes each) having buffer number 0 (buffer [0]). The buffer map may include buffer access information 433 for a device list buffer 450-1 (e.g., an array of PD identifiers) having buffer number 1 (buffer [1]). The buffer map 430 may include buffer access information 438 for a failed command number buffer 450-(n-1) having buffer number (n-1). The buffer map 430 may include buffer access information 439 for a status buffer 450-n having buffer number n. The buffer map 430 may include buffer access information 434 - 2 , . . . , 434 - n for buffer [ 2 ] , . . . , buffer [ n ] , respectively.

[0084] In some embodiments, the buffer map 430 may contain mapping information. For example, the buffer map 430 may specify a mapping between a command number (of a command) and a buffer number of an input buffer (corresponding to the command). The buffer map may specify a mapping between a command number (of a command) and a buffer number of an output buffer (corresponding to the command). For example, the buffer map 430 may contain command mapping information 436-0, ..., 436-m for command 0, ..., command m, each command mapping information indicating (1) a buffer number of an input buffer corresponding to the corresponding command and (2) a buffer number of an output buffer corresponding to the corresponding command.

[0085] In some embodiments, the buffer map 430 may contain information of one or more segment length buffers, such as the buffer number and the use of each segment length buffer. For example, if a segment length buffer is used for a particular buffer, the buffer access information for the particular buffer may include (1) a bit indicating the use of the segment length buffer, and (2) the buffer number of the segment length buffer to be used.

[0086] Figure 5 1 is a flow chart showing a process 500 for executing multiple commands in one or more storage devices according to an embodiment. In some embodiments, process 500 is performed by one or more processors (e.g., a processor of host 110, storage controller 130, or PD 150). In other embodiments, process 500 is performed by other entities. In some embodiments, process 500 includes more than Figure 5 More, fewer, or different steps may be shown in .

[0087] At step 502, one or more processors may receive a frame (e.g., request frame 310) that includes a number of commands (e.g., number of commands 327), a number of storage devices (e.g., number of PDs 328), and a buffer map (e.g., buffer maps 330, 430).

[0088] At step 504, in response to the frame, the one or more processors may use the buffer map and read from a memory (e.g., host memory 340, 440) (1) input data for each of the one or more storage devices corresponding to the number of storage devices 328 and (2) an identifier for each of the one or more storage devices (e.g., PD ID 0, ..., PD ID k specified in device list buffer 450[1]). In some embodiments, the one or more storage devices (e.g., PD 0, ..., PD k) may form a RAID device.

[0089] In some embodiments, the memory may include multiple buffers (e.g., buffer [0], ..., buffer [n]), and the buffer map 330, 430 may identify the location (e.g., offset in the host buffer 442) and length of each of the multiple buffers. Multiple commands (e.g., command 0, ..., command m) may be stored in a first buffer (PT command buffer 450[0]) of the multiple buffers of the memory in execution order. In some embodiments, the location may indicate an address or offset in the memory, or indicate any location information for locating a buffer in the memory (e.g., a memory address containing an address or offset of a buffer). In some embodiments, the length may indicate an arbitrary number of bits, bytes, kilobytes, etc. In some embodiments, identifiers of one or more storage devices (e.g., PD ID 0, ..., PD ID k) may be stored in a second buffer (e.g., device list buffer 450[1]) of the multiple buffers. Input data of the one or more storage devices may be stored in a third buffer (e.g., input buffer) of the multiple buffers. In some embodiments, the order in which identifiers of one or more storage devices are stored in the second buffer (e.g., in device list order) may be the same as the order in which input data of one or more storage devices are stored in the third buffer (e.g., in device list order).

[0090] At step 506, the one or more processors may send a plurality of commands corresponding to the number of commands to the one or more storage devices based at least on the input data of each storage device and the identifier of each storage device. In some embodiments, the plurality of commands may be SCSI pass-through commands and may be sent as frames according to MPI.

[0091] One or more processors may read a plurality of commands (e.g., command 0, ..., command m) from a first buffer (e.g., PT command buffer 450-0) and send the plurality of commands to one or more storage devices (e.g., PD 0, ..., PD k) in an execution order (e.g., the order of command 0, ..., command m). In some embodiments, when sending the plurality of commands, the one or more processors may send a first command (e.g., command 0) of the plurality of commands to the one or more storage devices. In response to sending the first command, the one or more processors may send a second command (e.g., command 1) of the plurality of commands to the one or more storage devices without waiting for the completion of the execution of the first command (e.g., command 0) on the one or more storage devices (e.g., PD 0, PD 1, ..., PD k). In some embodiments, the order in which the commands are embedded in the PT command buffer may be different from the order in which the commands are sent and executed. For example, when the plurality of commands are embedded in the order of command 0, command 1, ..., command m, the plurality of commands may be sent or executed in the order of, for example, command 1, command 4, command 0, command m, etc.

[0092] The term "coupled" and its variations include two components directly or indirectly joined to each other. The term "communicatively coupled" and its variations may include direct or indirect communicative coupling between two components. Such communication or communicative coupling may be achieved by a first component directly communicating with or directly coupled to a second component, or by an additional component interposed between the first and second components such that the first component indirectly communicates with or is indirectly coupled to the second component via the additional component.

[0093] The foregoing summarizes the features of several embodiments so that those skilled in the art can better understand aspects of the present disclosure. Those skilled in the art will appreciate that they can easily use the present disclosure as a basis for designing or modifying other processes and structures for achieving the same purposes and / or achieving the same advantages of the embodiments described herein. Those skilled in the art will also appreciate that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that they may make various changes, substitutions, and modifications herein without departing from the spirit and scope of the present disclosure.

[0094] It should be noted that certain paragraphs of the present disclosure may refer to terms associated with a subset of buffers, hosts, and devices, such as "first" and "second", for the purpose of identifying or distinguishing one from another or otherwise. These terms are not intended to relate entities (e.g., first device and second device) only in time or according to a sequence, although in some cases, these entities may contain such a relationship. These terms also do not limit the number of possible entities that can operate within a system or environment. It should be understood that the system described above may provide multiple of any or each of those components, and these components may be provided on a stand-alone machine, or in some embodiments, on multiple machines in a distributed system. In addition, the above-described systems and methods may be provided as one or more computer-readable programs or executable instructions embodied on or in one or more articles of manufacture (e.g., floppy disks, hard disks, CD-ROMs, flash memory cards, PROMs, RAMs, ROMs, or tapes). The program may be implemented in any programming language (e.g., LISP, PERL, C, C++, C#) or in any byte code language (e.g., JAVA). The software program or executable instructions may be stored as object code on or in one or more articles of manufacture.

[0095] Although the foregoing written description of the methods and systems enables one of ordinary skill in the art to make and use embodiments thereof, one of ordinary skill in the art will understand and appreciate the existence of variations, combinations, and equivalents of the specific embodiments, methods, and examples herein. Therefore, the methods and systems should not be limited to the embodiments, methods, and examples described above, but should be limited to all embodiments and methods within the scope and spirit of the present disclosure.

Claims

1. A system comprising: One or more processors configured to: receiving a frame, the frame comprising a number of commands, a number of storage devices, and a buffer map; Responsive to the frame, using the buffer map and reading from a memory (1) input data for each of one or more storage devices corresponding to the number of storage devices, and (2) an identifier for each of the one or more storage devices; and A plurality of commands corresponding to the number of commands is sent to the one or more storage devices based at least on the input data of each storage device and the identifier of each storage device.

2. The system according to claim 1, wherein The memory includes a plurality of buffers, and The buffer map identifies the location and length of each of the plurality of buffers.

3. The system according to claim 2, wherein The plurality of commands are stored in a first buffer of the plurality of buffers in an execution order, and The one or more processors are configured to The plurality of commands are read from the first buffer and sent to the one or more storage devices in the execution order.

4. The system according to claim 2, wherein The identifier of the one or more storage devices is stored in a second buffer of the plurality of buffers, and The input data of the one or more storage devices is stored in a third buffer among the plurality of buffers.

5. The system of claim 1 , wherein in sending the plurality of commands, the one or more processors are configured to: sending a first command of the plurality of commands to the one or more storage devices; and In response to sending the first command, a second command of the plurality of commands is sent to the one or more storage devices without waiting for completion of execution of the first command on the one or more storage devices.

6. The system of claim 1, wherein The one or more storage devices form a redundant array of independent disks RAID device.

7. The system of claim 1, wherein The plurality of commands are small computer system interface SCSI pass-through commands and are sent as frames according to a message passing interface MPI.

8. A device comprising: Memory; and A controller coupled to a plurality of storage devices, wherein The controller is configured to receiving a frame including a number of commands, a number of storage devices among the plurality of storage devices, and a buffer map, responsive to the frame, using the buffer map and reading from the memory (1) input data for each of one or more storage devices corresponding to the number of storage devices, and (2) an identifier for each of the one or more storage devices, and A plurality of commands corresponding to the number of commands is sent to the one or more storage devices based at least on the input data of each storage device and the identifier of each storage device.

9. The device according to claim 8, wherein The memory includes a plurality of buffers, and The buffer map identifies the location and length of each of the plurality of buffers.

10. The device according to claim 9, wherein The plurality of commands are stored in a first buffer of the plurality of buffers in an execution order, and The controller is configured to The plurality of commands are read from the first buffer and sent to the one or more storage devices in the execution order.

11. The device according to claim 9, wherein The identifier of the one or more storage devices is stored in a second buffer of the plurality of buffers, and The input data of the one or more storage devices is stored in a third buffer among the plurality of buffers.

12. The apparatus of claim 8, wherein in sending the plurality of commands, the controller is configured to: sending a first command of the plurality of commands to the one or more storage devices; and In response to sending the first command, a second command of the plurality of commands is sent to the one or more storage devices without waiting for completion of execution of the first command on the one or more storage devices.

13. The device according to claim 8, wherein The one or more storage devices form a redundant array of independent disks RAID device.

14. The device according to claim 8, wherein The plurality of commands are small computer system interface SCSI pass-through commands and are sent as frames according to a message passing interface MPI.

15. A method comprising: receiving, by one or more processors, a frame including a number of commands, a number of storage devices, and a buffer map; In response to the frame, read, by the one or more processors, using the buffer map and from a memory (1) input data for each of the one or more storage devices corresponding to the number of storage devices, and (2) an identifier for each of the one or more storage devices; and A plurality of commands corresponding to the number of commands is sent, by the one or more processors, to the one or more storage devices based at least on the input data of each storage device and the identifier of each storage device.

16. The method according to claim 15, wherein The memory includes a plurality of buffers, and The buffer map identifies the location and length of each of the plurality of buffers.

17. The method according to claim 16, wherein The plurality of commands are stored in a first buffer of the plurality of buffers in an execution order, and The method comprises: The plurality of commands are read from the first buffer and sent to the one or more storage devices in the execution order.

18. The method according to claim 16, wherein The identifier of the one or more storage devices is stored in a second buffer of the plurality of buffers, and The input data of the one or more storage devices is stored in a third buffer among the plurality of buffers.

19. The method of claim 15, wherein sending the plurality of commands comprises: sending a first command of the plurality of commands to the one or more storage devices; and In response to sending the first command, a second command of the plurality of commands is sent to the one or more storage devices without waiting for completion of execution of the first command on the one or more storage devices.

20. The method of claim 15, wherein The one or more storage devices form a redundant array of independent disks RAID device.

Images