Method and system for issuing io between controller and disk across sas expander

Abstract

This invention provides a method and system for distributing I / O across SAS expanders between a controller and a disk. The method includes: an ITS module retrieving I / O requests from an I / O request queue and encapsulating them into SCSI commands; selecting a target device address; selecting the local expander channel address for odd-numbered slots and the peer expander channel address for even-numbered slots; the SAS driver transmitting the SCSI command to the local expander via the SAS bus; the local expander selecting the corresponding channel for transmission based on the SAS address encapsulated in the SCSI command; and when the local expander fails to distribute the I / O, it returns a corresponding I / O error to the ITS module, which modifies the SAS address in the SCSI command to the corresponding channel address on the peer expander and re-distributes the I / O. This invention provides a method and system for distributing I / O across SAS expanders between a controller and a disk, allowing I / O to be distributed to the storage disk via the expander belonging to the peer controller when I / O distribution between the current controller and the expander / storage disk fails.

Description

Technical Field

[0001] This invention relates to the field of storage system technology, and in particular to a method and system for distributing I / O across a SAS expander between a controller and a disk. Background Technology

[0002] Since the beginning of the 21st century, with the rise of the Internet of Things and big data, society is entering the era of information and big data. While this era brings convenience, it also presents numerous challenges. Ensuring the security and efficiency of data storage is a common problem facing the storage industry. Currently, common underlying storage control system architectures mainly consist of controllers and disk enclosures. The data interaction path between the controller and the disks largely determines the data transmission capability of the storage system. A reliable I / O interaction path must ensure the stability of I / O transmission while maximizing the I / O transmission rate.

[0003] In most storage systems, to improve reliability and prevent data loss due to storage controller failure during I / O data transmission, a dual-controller architecture is often used. This involves redundant data exchange between two controllers to complete the corresponding storage services. In a dual-controller storage system, both controllers share the right to use the storage disks on the storage control cabinet. Currently, most common storage disks are SAS or SATA types. SAS (Serial Attached SCSI) networks support both SAS and SATA disks, so the transmission network between the storage controller and storage devices is typically implemented using a SAS network. To increase the capacity of the storage devices connected to the controller, SAS expanders (storage expansion cabinets) are usually used to expand the number of storage devices in the storage system, forming a transmission network between the controller, expanders, and storage devices.

[0004] In a dual-controller storage array, the controllers and disk enclosures are connected via SAS expanders (disk output expanders). Each controller is connected to one output expander, and the controllers and expanders, as well as the expanders and disk enclosures, are connected via SAS cables. A SAS expander can be viewed as a hard drive switch, allowing multiple hard drives to be connected together. SAS expanders are divided into fan-out expanders and edge expanders, with fan-out expanders connecting to edge expanders via topology expanders. Expanders interact with devices via transmission ports, which are divided into wide ports and narrow ports. A single wide port contains multiple PHY implementations, while a narrow port consists of a single PHY. Each PHY is a differential signal pair consisting of a transmitter and a receiver. Each PHY has a SAS address and a unique PHY ID. PHYs within the same transmission port have the same SAS address. The transmission rate of a wide port is obtained by summing the rates of each PHY within the port. The channel between the expander and the disk is a narrow channel, while the connection channel between expanders is a wide channel. The bandwidth and stability of the transmission channel determine the performance of I / O.

[0005] See Figure 1 In existing storage systems, I / O communication between the controller and the disk is usually conducted through the PHY channel under the local expander, or forwarded to the other end controller through the communication mechanism between controllers. Currently, there are many patents in the industry on high-speed communication mechanisms between controllers, but few patents on expander I / O transmission links between controllers, which is worth further study. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method and system for distributing I / O across SAS expanders between a controller and a disk. When I / O distribution between the current controller and the expander or storage disk fails, I / O can be distributed to the storage disk through the expander of the other controller.

[0007] To address the aforementioned technical problems, this invention provides a method for issuing I / O across a SAS expander between a controller and a disk. The method includes: issuing an I / O operation; the SCHED module encapsulating the I / O operation into an I / O request frame and inserting it into an I / O request queue; the ITS module retrieving the I / O request from the I / O request queue, encapsulating it into a SCSI command, selecting the target device address (odd slots select the local expander channel address, even slots select the peer expander channel address); the SAS driver transmitting the SCSI command to the local expander via the SAS bus; the local expander selecting the corresponding channel for transmission based on the SAS address encapsulated in the SCSI command; when the local expander I / O issuance fails due to a busy PHY link or connection error, a corresponding I / O error is returned to the ITS module, which modifies the SAS address in the SCSI command to the corresponding channel address on the peer expander and reissues the I / O.

[0008] In some implementations, the SCHED module encapsulates the IO operation into an IO request frame and inserts it into the IO request queue. This includes: the SCHED module encapsulating the IO operation into an IO request frame and forwarding it to the ITS module; the ITS module placing the IO request into the IO request queue and returning, waiting for the ITS module to process it.

[0009] In some implementations, the ITS module retrieves I / O requests from the I / O request queue, encapsulates them into SCSI commands, selects the target device address, and includes: the ITS module completing initialization by creating a global thread.

[0010] In some implementations, a global thread is responsible for retrieving IO requests one by one from the IO request queue, checking whether the IO requests are valid, returning IO errors for invalid IO requests, and initializing valid IO requests into SCSI command format.

[0011] In some implementations, the target SAS address of the SCSI command frame is initialized based on the disk slot id.

[0012] In some implementations, the target SAS address of the SCSI command frame is initialized based on the disk slot ID, including: the target SAS address for odd-numbered disk I / O requests is initialized to the SAS address of the PHY channel on the local expander, and the target SAS address for even-numbered disk I / O requests is initialized to the PHY channel address on the remote expander.

[0013] In some implementations, the local expander selects the appropriate channel for transmission based on the SAS address encapsulated in the SCSI command. This includes: the underlying driver transmitting the SCSI command to the local SAS expander via the SAS bus; the expander routing mechanism determining whether to use the local PHY channel or the PHY channel of the remote expander disk based on the target SAS address; if the target SAS address is remote, then the IO SCSI frame is forwarded to the remote expander via the wide port between the two controllers' expanders, and sent to the disk via the PHY channel, thus distributing the IO to the corresponding disk device.

[0014] Furthermore, this invention also provides a cross-SAS expander I / O system between a controller and a disk, comprising: a controller, a disk enclosure, and an expander. The controller and the disk enclosure are connected via an output expander, and each controller is connected to an output expander. The controller and the expander, and the expander and the disk enclosure, are connected via SAS cables. The expander includes a local expander and a remote expander. The controller, disk enclosure, and expander implement the cross-SAS expander I / O method between the controller and the disk as described above.

[0015] With this design, the present invention has at least the following advantages:

[0016] By introducing a mechanism for multi-path IO distribution through the link relationship between expanders, the fault tolerance of the storage system is improved, the transmission pressure on the controller expander is reduced, the load balancing effect is achieved, and the product competitiveness is enhanced. Attached Figure Description

[0017] The above is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Figure 1 This is a system architecture diagram of the existing storage system;

[0019] Figure 2 This is a flowchart of the method for issuing I / O across the SAS expander between the controller and the disk provided in an embodiment of the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] The embodiments of the present invention include the following operation steps:

[0022] Business logic issues I / O operations.

[0023] The SCHED module encapsulates I / O operations into I / O request frames and inserts them into the I / O request queue.

[0024] The ITS module retrieves I / O requests from the I / O request queue, encapsulates them into SCSI commands, selects the target device address, selects the local expander channel address for odd-numbered slots, and selects the remote expander channel address for even-numbered slots.

[0025] The SAS driver transmits SCSI commands to the local expander via the SAS bus.

[0026] The local expander selects the appropriate channel for transmission based on the SAS address encapsulated in the SCSI command.

[0027] When the local expander IO fails to be issued due to a busy PHY link or connection error, the corresponding IO error is returned to the ITS module. The ITS module then modifies the SAS address in the SCSI command to the corresponding channel address on the peer expander and reissues the IO.

[0028] By adopting the above technical solution, an I / O multi-path delivery mechanism was introduced through the link relationship between expanders, which improved the fault tolerance of the storage system, reduced the transmission pressure on the expander controller, achieved load balancing, and improved product competitiveness.

[0029] See Figure 2 This section contains the following parts:

[0030] Step 1: After the business initiates an IO operation, SCHED (data forwarding and scheduling module) encapsulates the IO operation into an IO request frame and forwards it to the ITS (Initiator Transport Subsystem) module. The ITS module puts the IO request into the IO request queue and returns, waiting for the ITS module to process it.

[0031] Step 2: During initialization, the ITS module creates a global thread responsible for retrieving IO requests one by one from the IO request queue, checking whether the IO requests are valid, returning IO errors for invalid IO requests, and initializing valid IO requests into SCSI (Small Computer System Interface) command format. The target SAS address of the SCSI command frame is initialized according to the disk slot ID. For odd-numbered disks, the target SAS address of the IO request is initialized to the SAS address of the PHY channel on the local expander, and for even-numbered disks, the target SAS address is initialized to the PHY channel address on the remote expander. The initialized SCSI command is then inserted into the disk device's SCSI pending queue.

[0032] Step 3: When the ITS module's SCSI command processing logic reaches the disk device, it retrieves the SCSI command from the disk device's SCSI pending queue, calls the underlying SAS driver's IO command execution function, and processes the SCSI command.

[0033] Step 4: The underlying driver transmits SCSI commands to the local SAS expander via the SAS bus. The expander routing mechanism determines whether to use the local PHY channel or the remote expander's disk PHY channel based on the target SAS address. If the target SAS address is remote, the I / O SCSI frame is forwarded to the remote expander via the wide port between the two controllers' expanders, and then sent to the disk via the PHY channel, thus distributing the I / O to the corresponding disk device. For successfully executed commands, a success message is returned; for commands that fail, the error handling logic is entered.

[0034] Step 5: When the ITS module receives a no-response or failure return for disk I / O, and still fails after 3 retries, it reinitializes the SCSI I / O command, modifies the target SAS address to the peer expander PHY channel address, and resends the SCSI command. The local expander routing mechanism selects the wide channel address connected to the peer expander when choosing the target address, and sends it to the peer expander. The peer expander then resends the command to the storage disk through the PHY channel between the peer expander and the disk. This improves system fault tolerance and prevents single points of failure.

[0035] This invention also provides a system for distributing I / O across a SAS expander between a controller and a disk. The system includes a controller, a disk enclosure, and an expander. The controller and the disk enclosure are connected via expanders. Each controller is connected to an output expander. The controller and the expander, and the expander and the disk enclosure, are connected via SAS cables. The expander includes a local expander and a remote expander. The controller, disk enclosure, and expander implement the method for distributing I / O across a SAS expander between the controller and the disk as described above.

[0036] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, or alterations made by those skilled in the art using the disclosed technical content shall fall within the protection scope of the present invention.

Claims

1. A method for issuing I / O across a SAS expander between a controller and a disk, characterized in that, include: Business logic issues I / O operations; The SCHED module encapsulates I / O operations into I / O request frames and inserts them into the I / O request queue; The ITS module retrieves I / O requests from the I / O request queue, encapsulates them into SCSI commands, selects the target device address, selects the local expander channel address for odd-numbered slots, and selects the remote expander channel address for even-numbered slots. The SAS driver transmits SCSI commands to the local expander via the SAS bus; The local expander selects the appropriate channel for transmission based on the SAS address encapsulated in the SCSI command, including: The underlying driver transmits SCSI commands to the local SAS expander via the SAS bus. The expander routing mechanism determines whether to use the local PHY channel or the PHY channel of the expander disk on the other end based on the target SAS address. If the target SAS address is on the other end, the IO SCSI frame is forwarded to the expander on the other end through the wide port between the expanders of the two controllers, and sent to the disk through the PHY channel, thus distributing the IO to the corresponding disk device. When the local expander IO fails to be issued due to a busy PHY link or connection error, the local expander returns the corresponding IO error to the ITS module. The ITS module then modifies the SAS address in the SCSI command to the corresponding channel address on the peer expander and reissues the IO.

2. The method for issuing I / O across SAS expanders between the controller and the disk according to claim 1, characterized in that, The SCHED module encapsulates I / O operations into I / O request frames and inserts them into the I / O request queue, including: The SCHED module encapsulates the IO operation into an IO request frame and forwards it to the ITS module. The ITS module puts the IO request into the IO request queue and returns, waiting for the ITS module to process it.

3. The method for issuing I / O across SAS expanders between the controller and the disk according to claim 1, characterized in that, The ITS module retrieves I / O requests from the I / O request queue, encapsulates them into SCSI commands, and selects the target device address, including: The ITS module completes its initialization by creating a global thread.

4. The method for issuing I / O across SAS expanders between the controller and the disk according to claim 3, characterized in that, The global thread is responsible for retrieving IO requests one by one from the IO request queue, checking whether the IO request is valid, returning an IO error for invalid IO requests, and initializing valid IO requests into SCSI command format.

5. The method for issuing I / O across SAS expanders between the controller and the disk according to claim 4, characterized in that, The target SAS address of the SCSI command frame is initialized based on the disk slot id.

6. The method for issuing I / O across SAS expanders between the controller and the disk according to claim 5, characterized in that, The target SAS address of the SCSI command frame is initialized based on the disk slot ID, including: For odd-numbered disks, the target SAS address for I / O requests is initialized to the SAS address of the PHY channel on the local expander, while for even-numbered disks, the target SAS address for I / O requests is initialized to the PHY channel address on the remote expander.

7. A cross-SAS expander I / O system between a controller and a disk, characterized in that, include: The controller, disk control cabinet, and expander are connected via expanders. Each controller is connected to an output expander. The controller and expander, and the expander and disk control cabinet are connected via SAS cables. The expander includes a local expander and a remote expander. The controller, disk control cabinet, and expander implement the method for issuing I / O across the SAS expander between the controller and the disk as described in any one of claims 1 to 6.

Images