Restoring connection speed in a data storage system

Abstract

Link speed recovery in a data storage system according to the present description, in one aspect, comprises repeating the execution of a main loop of consecutive link speed recovery instructions a predetermined maximum number of times. In one embodiment, the main loop execution of link speed recovery instructions comprises repeating the execution of a sub-loop of consecutive link speed recovery instructions within each main loop execution a predetermined maximum number of times.As a result of repeating the execution of a sub-loop of consecutive link speed recovery commands within each main loop execution and repeating the execution of the main loop of consecutive link speed recovery commands according to one embodiment, the reliability of link speed recovery to full link speed can be improved. Depending on the particular use case, other aspects and advantages may be realized.

Description

BACKGROUND OF THE INVENTION AREA OF THE INVENTION

[0001] The present invention relates to a computer program product, a system and a method for restoring connection speed in a data storage system. DESCRIPTION OF THE STATE OF THE ART

[0002] A storage system typically comprises a storage control unit and one or more data storage units, such as hard disk drives, semiconductor memory, tape drives, etc. The storage system is often connected to a host that hosts applications which issue input / output instructions or commands to write data to or read data from a storage subunit, such as a data storage medium.

[0003] Data storage systems, especially at the enterprise level, are typically designed to provide a high degree of redundancy to reduce the risk of data loss in the event of a system component failure. Therefore, multiple copies of data are often stored on multiple systems, which may be geographically distributed. Data from a host destined for the data storage system is usually directed to a primary data storage unit at a local location and then replicated to one or more secondary data storage units, which may be geographically distant from the primary data storage unit.

[0004] In certain data processing environments, a storage network provides data transmission paths or channels between multiple host systems and multiple storage control units, which in turn control multiple storage control units, such as a Direct Access Storage Device (DASD), a Redundant Array of Independent Disks (RAID), a Just a Bunch of Disks (JBOD), etc. The data transmission paths across the storage network typically include switches and data connections, which may consist of fiber optic cables or other types of cables, or be wireless.

[0005] Input / output commands issued by a host are routed through the storage network to a port on a storage control unit (SCU). Each SCU port typically has an address or other identifier to distinguish it from other ports on the storage network. In response to I / O commands from a host, the SCU reads data from or writes data to storage devices connected to ports of one or more SCUs via data paths on the storage network. The data paths connecting the SCUs to the storage devices, such as those connecting the host and SCUs, typically include switches and data links, which may be constructed from fiber optic cables or other types of cables, or they may be wireless.

[0006] The data transport capacity of a storage network data path is typically limited by several factors. For example, storage network data paths are often designed or configured to support predetermined data transfer rates, measured in a specific number of data units such as bytes or bits, for instance, per second. Accordingly, the data transport capacity of a given connection is generally limited by the connection speed of the data transmission link.

[0007] In a serial bus data transmission path, such as a connection or channel according to Peripheral Component Interconnect Express (PCIe), a data transmission link can contain multiple lanes along which data can move between an upstream unit, such as a switch, and a downstream unit, such as an endpoint device. Therefore, the link speed depends on the data transfer rate of a lane and the number of lanes allocated to the link. This allocation is negotiated by upstream and downstream control units of the path in a process often referred to as "training" or "retraining" the link.

[0008] For example, when a switch in the form of a PCIe add-in card is plugged into a data path host (root complex) of a storage control unit, upstream and downstream control units of the link connecting the storage control unit to a storage unit exchange "training sequences" to negotiate various link parameters, including the data transfer rate. This is achieved by running a link training and status state machine (LTSSM), which typically initiates the link by establishing a single lane that provides a relatively low data transfer rate.However, as lanes are added through the link training process, the link speed available for data transfer increases until a target link speed is reached, referred to herein as "full speed" or "full link speed".

[0009] After the training process is complete, data can be transferred at the increased data transfer rate achieved through the training. However, the connection training process may not have achieved the full speed or the target speed for the connection. Even if the full speed was achieved through connection training, data transmission conditions may change, requiring the data transmission path to renegotiate transmission parameters. This can "downtrain" the connection speed to a lower speed than the optimal or target speed. Furthermore, data transmission may be lost over some or all of the connection's lanes due to various factors.

[0010] If the connection is not operational or is operating at a lower than full speed, the data transmission path can renegotiate link parameters to retrain the connection, hopefully achieving or restoring full speed. However, it is understood that such attempts to retrain or retrain connections are often unsuccessful.

[0011] For example, the PCIe specification provides a data path host, typically a root complex, to initiate link retraining by setting a configuration bit of the PCI Express Capability Link Status Control register of the upstream or downstream control unit. However, if no link lanes are currently established, the root complex cannot exchange data with the downstream control unit about the link to set the downstream control unit's configuration bit. Furthermore, setting a configuration bit of a link control unit's PCI Express Capability Link Status Control register when initiating link retraining to restore full link speed is often unsuccessful.

[0012] A common technique for initiating link training involves the root complex resetting or disabling / enabling a hardware component of a PCIe data transmission path. For example, an optical transceiver of a PCIe port can be reset, or the entire PCIe card, which may have one or more PCIe ports, can be reset. Another example is toggling a PCIe port off and on again in an enable / disable cycle to initiate training or retraining of the data transmission link. If no lanes of the link are currently configured, the root complex cannot exchange data with the downstream control unit over the link to reset the downstream component.Furthermore, resetting or turning a PCIe component off and on again when initiating a connection retraining process to restore full connection speed is often unsuccessful.

[0013] The publication DE 10 2018 005 753 A1 concerns a method for performing connection training for one or more input / output connections that connect an upstream port to a downstream port.The procedure comprises: storing one or more matching parameter values ​​for the downstream port in non-volatile memory associated with the upstream port; performing an initialization sequence of the one or more input / output links between the upstream port and the downstream port, wherein the initialization sequence includes link training of the one or more input / output links; retrieving the stored matching parameter values ​​for the downstream port from the non-volatile memory; writing the stored matching parameter values ​​for the downstream port to a register associated with the downstream port; and using the matching parameter values ​​as matching parameters for the downstream port to operate the one or more links connecting the upstream port to the downstream port.

[0014] Document US 2014 / 0006675A1 relates to a PCIe interface module and a physical layer used to negotiate a link by exchanging a number of fast training sequences (FTS). The physical layer can count the number of good FTS exchanged during an initial or subsequent link training session. The number of FTS exchanged during a subsequent link training session can be a number where the maximum initial number of fast training sequences to be exchanged is reduced by the number of good FTS exchanged during the initial link training session, thereby reducing the link training time and increasing efficiency.

[0015] Document US 2013 / 0051483A1 concerns a system. The system comprises a first device, a second device, a communication link, and a memory. The memory stores instructions which, when executed by the system, perform a procedure for training the communication link. This procedure involves requesting a speed change to a second speed for the first device, which communicates with the second device at a first speed via the communication link. It accesses a stored set of parameters for at least one of the two devices, the first and the second. An initial training cycle is performed for the first device and the second device at the second speed using the stored parameter set for the first device and / or the second device.By reusing parameters from a previous successful training cycle for equalization, the time required to perform the equalization training is reduced.

[0016] Publication US 2014 / 0108686A1 relates to a method. The method comprises: In a first integrated circuit connected via a physical link to a second integrated circuit, executing a detection state of a link training state machine of a physical unit (PHY) of a low-energy communication protocol, including a physical circuit unit, wherein the PHY unit is connected to a protocol stack for a communication protocol for a Peripheral Component Interconnect Express™ (PCIe™) communication protocol, including a transaction layer and data link layer, in response to the undervoltage setting of the first integrated circuit; and executing, in the first integrated circuit, a configuration state of the link training state machine after the execution of the detection state.including sending a config ready signal to the second integrated circuit via a sideband link between the first and second integrated circuits, and executing, in the first integrated circuit, a stall state of the link training state machine in response to receiving a second config ready signal from the second integrated circuit via the sideband link, wherein the PHY unit controls a differential N signal on the physical link during the stall state. SUMMARY

[0017] The invention is based on the objective of providing a method, a system, a computer program, a computer program product, and a computer system for efficiently restoring connection speed in a data storage system. This objective has been achieved by the features of the independent claims. Preferred embodiments are specified in the dependent claims.

[0018] Restoring link speed in data storage systems, as described here, represents a significant improvement in computer technology. In one aspect, performing link speed restoration to achieve a target link speed on a data transmission link involves repeating a main loop of successive link speed restoration commands a predetermined maximum number of times. Furthermore, each execution of link speed restoration commands in the main loop includes repeating a sub-loop of successive link speed restoration commands within each main loop, again with a predetermined maximum number of repetitions.

[0019] In one embodiment, executing a set of consecutive commands for the sub-loop of consecutive link-speed restoration commands may be more successful than executing a set of consecutive commands for the main loop. Therefore, repeating the sub-loop's set of consecutive link-speed restoration commands within each main loop, and thus more frequently than executing the main-loop's set of consecutive link-speed restoration commands, may improve the success rate of training or retraining the link to achieve or restore full link speed.

[0020] In another aspect, each execution of the sub-loop of successive link speed restoration commands within each main loop execution can optionally include issuing a next link speed restoration command of the sub-loop's set of successive link speed restoration commands, determining whether the target link speed has been reached after issuing a next link speed restoration command, and stopping the link speed restoration execution if the target link speed has been reached in response to a next link speed restoration command.Furthermore, each main loop execution of link speed recovery commands can optionally include, after repeating the execution of the sub-loop of link speed recovery commands a specified maximum number of times, issuing a next link speed recovery command of the main loop set of successive link speed recovery commands for the main loop, determining whether the target link speed has been reached after issuing a next link speed recovery command, and terminating the link speed recovery execution if the target link speed has been reached in response to a next link speed recovery command.

[0021] It is understood that executing a specific command to restore connection speed may not result in the target connection speed being reached. By determining whether the target speed has been reached after issuing a subsequent command to restore connection speed, the restoration process can be stopped as soon as the target speed is achieved, thus improving the efficiency of the connection speed restoration process.

[0022] Conversely, if it is determined that the target speed has not been restored, another subsequent link speed restoration command can be issued immediately, following a set of consecutive link speed restoration commands, to continue link speed restoration until the target speed is restored. Accordingly, in one aspect, each execution of the sub-loop of link speed restoration commands can optionally involve issuing another subsequent link speed restoration command from the sub-loop's set of link speed restoration commands.If, in response to a link speed recovery command of the child loop set, the target link speed is not reached, the execution may include determining whether the target link speed has been reached after issuing a subsequent link speed recovery command, and terminating the link speed recovery execution if the target link speed has been reached in response to another subsequent link speed recovery command of the child loop set. Furthermore, each main loop execution of link speed recovery commands may optionally include issuing another subsequent link speed recovery command of the main loop set of link speed recovery commands.If, in response to a command to restore the link speed of the main loop set, the target link speed is not reached, the process should include determining whether the target link speed has been reached after issuing a subsequent command to restore the link speed, and terminating the link speed restoration process if the target link speed has been reached in response to a further subsequent command to restore the link speed of the main loop set.

[0023] Another aspect is that each main loop execution of link speed restoration commands can optionally include termination after repeating the main link speed restoration command loop a predetermined maximum number of times if the target link speed has not been achieved in response to a link speed restoration command. It is understood that limiting the number of main loop executions when the full link speed has not been reached within these limits can improve the allocation of computer system resources.However, other techniques can be used, such as replacing defective components and repeating the restoration of the connection speed as described herein.

[0024] In yet another aspect, the set of successive instructions for restoring the link speed of the sub-loop can optionally include at least one of setting a configuration bit of a register on an upstream port connected to the data link, sending an out-of-band signal to set a configuration bit of a register on a downstream port connected to the data link, starting a disable-enable cycle of the downstream port, and sending an out-of-band signal to start a disable-enable cycle of the downstream port.It is understood that executing this set of consecutive commands for the child loop of consecutive commands for restoring connection speed may be more frequently successful than executing a different set of consecutive commands for the main loop. Therefore, repeating this set of consecutive commands for the child loop of consecutive commands for restoring connection speed more frequently than repeating the set of consecutive commands for the main loop of consecutive commands for restoring connection speed may improve the success rate with which retraining the connection leads to a restoration of full connection speed.

[0025] In yet another aspect, the main loop set of successive link speed restoration commands can optionally include at least one of the following: initiating the sub-loop of successive link speed restoration commands, initiating a reset and reinitialization of a data transmission path switch that includes an upstream port containing an upstream optical transceiver connected to the data transmission link, resetting the upstream optical transceiver, and sending an out-of-band signal to reset a downstream optical transceiver connected to the data transmission link.It is understood that the execution of this set of consecutive commands for the subordinate loop of consecutive commands to restore the connection speed may be successful, while the execution of a set of consecutive commands for the main loop may not be successful.Accordingly, the set of consecutive commands for the main loop of consecutive link speed recovery commands adds additional recovery commands that provide other wide link speed recovery pathways to improve the success rate with which training or retraining the link results in achieving or restoring full link speed, although in some embodiments the set of consecutive commands for the main loop of consecutive link speed recovery commands may be executed less frequently than the repetition of the set of consecutive commands for the sub-loop of consecutive link speed recovery commands. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates an embodiment of a data processing environment in which the restoration of link speed in a data storage system is applied according to one aspect of the present description. Fig. Figure 2 illustrates an example of a data storage system that includes a storage control unit and a storage unit of the data processing environment. Fig. 1 includes, in which the restoration of the connection speed in a data storage system is applied according to one aspect of the present description. Fig. Figure 3 illustrates an example of a primary and a secondary data storage system, which includes storage control units of the data processing environment of Fig. 1 includes, in which the restoration of the connection speed in a data storage system is applied according to one aspect of the present description. Fig. Figure 4 illustrates an example of a data transfer path in the data processing environment of Fig. 1, where the restoration of connection speed in a data storage system is applied according to one aspect of the present description. Fig. Figure 5 illustrates an example of a host of the data processing environment of Fig. 1, where the restoration of connection speed in a data storage system is applied according to one aspect of the present description. Fig. Figure 6 illustrates an example of operations performed by components of the data processing environment of Fig. 1, where the restoration of connection speed in a data storage system is applied according to one aspect of the present description. The Fig. 7A and Fig. 7B illustrates exemplary sets of consecutive commands for loop operations of components of the data processing environment of Fig. 1, where the restoration of connection speed in a data storage system is applied according to one aspect of the present description. Fig. Figure 8 illustrates a computer implementation form in which the restoration of link speed in a data storage system is applied according to one aspect of the present description. DETAILED DESCRIPTION

[0026] Restoring connection speed in data storage systems, as described in this document, represents a significant improvement in computer technology. In one embodiment, the connection speed restoration operations are directed toward training a data transmission link of a data transmission path to achieve or regain a target connection speed, such as full connection speed. Accordingly, the term "training" a data transmission link, as used herein, also includes retraining a data transmission link.For example, link speed restoration as described herein can enable the retraining of a data transmission link after a loss of full link speed, where one or more lanes of the data transmission link were lost. Similarly, link speed restoration as described herein can enable the training of a data transmission link to achieve full link speed when a data transmission link is initialized for the first time. Accordingly, the term "link speed restoration," as used herein, refers to the training or retraining of a data transmission link to achieve a target link speed, such as full transmission speed on the first or subsequent occasions.

[0027] In one aspect, the restoration of connection speed as described herein involves executing a set of sequential connection speed restoration commands within a main loop. In one embodiment, the main loop, and thus the set of sequential connection speed restoration commands within the main loop, is repeated as needed. Furthermore, in one embodiment, the main loop includes a sub-loop containing another set of sequential connection speed restoration commands, which are also repeated as needed.Thus, within the execution of each main loop of a set of successive link speed restoration commands, a sub-loop of another set of successive link speed restoration commands is repeated as needed.

[0028] In one aspect of the present description, the set of consecutive commands for restoring the connection speed of the child loop is repeated more often than the set of consecutive commands for restoring the connection speed of the main loop. Accordingly, the child loop's set of commands for restoring the connection speed can be assigned different commands than the main loop's set to maximize the efficiency of the connection speed restoration.For example, it is assumed that by repeating the execution of the sub-loop set of consecutive link-speed restoration commands within each main-loop execution, and by repeating the execution of the main-loop set of consecutive link-speed restoration commands according to one embodiment, the reliability of training or retraining the link speed to full link speed can be improved. Accordingly, the success rate with which link training leads to a restoration of full link speed can also be improved. Depending on the specific use case, other aspects and benefits may be realized.

[0029] A system consisting of one or more computers can be configured to restore link speed according to the present description by installing software, firmware, hardware, or a combination thereof on the system that, when operational, causes the system to perform write transfer resource management as described herein. For example, one or more computer programs for restoring link speed in a data storage system can be configured to contain instructions that, when executed by a data processing device such as a processor of a storage control unit, cause the device to perform the actions.In the sense used herein, the term "compression" refers to any suitable algorithmic compression process that processes data by re-encoding a string of data such that the size of the resulting string of data is reduced compared to the size of the original string of data before algorithmic processing.

[0030] The operations described herein are performed by logic configured to be carried out either automatically or substantially automatically with little or no operator intervention, except where specified as being performed manually. Therefore, as used herein, the term "automatic" includes fully automatic operations, that is, operations performed by one or more hardware- or software-controlled machines without human intervention such as user input into a graphical user interface. As used herein, the term "automatic" further includes predominantly automatic operations, i.e.,, most of the operations (such as over 50%) are performed by one or more hardware- or software-controlled machines without human intervention such as user input into a graphical user selection interface, and the remaining operations (such as less than 50%) are performed manually, i.e., the manual operations are performed by one or more hardware- or software-controlled machines with human intervention such as user input into a graphical user selection interface.

[0031] Many of the functional elements described in this description are referred to as "logic elements" to emphasize the independence of their implementation. For example, a logic element can be implemented as a hardware circuit featuring custom VLSI circuits or gate arrays, commercially available semiconductors such as logic circuits, transistors, or other discrete components. A logic element can also be implemented as programmable hardware units, such as FPGAs (Field Programmable Gate Arrays), PALs (Programmable Array Logic), programmable logic units, or similar devices.

[0032] A logic element can also be implemented in software for execution on various types of processors. A logic element comprising executable code can, for example, contain one or more physical or logical blocks of computer instructions, which may be organized as objects, procedures, or functions. Regardless, the executable components of a designated logic element need not be physically located together; they can consist of entirely different instructions stored in different locations. When combined, these instructions constitute the logic element and achieve its intended purpose.

[0033] In fact, executable code for a logic element can consist of a single instruction or many instructions, which may even be distributed across multiple different code segments, different programs, different processors, and multiple memory units. Likewise, operational data, described and illustrated herein, can reside within logic elements and be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be captured as a single data record, or it may be distributed across different memory locations, including different memory units.

[0034] The Fig. Figures 1 to 5 illustrate an embodiment of a data processing environment in which link speed restoration is used in a data storage system as described herein. In this example, a plurality of hosts 1a, 1b...1n can transmit input / output (I / O) requests over a network 6a to one or more storage control units 4a, 4b...4n to access data stored by the storage control units 4a, 4b...4n over a network 6b in a storage device 10a, 10b...10n. Each storage control unit and each storage device controlled by the storage control unit over the network 6b constitutes a data storage system. The hosts 1a, 1b...1n can be separate physical units or virtual units, implemented, for example, using allocated resources from partitions of a server.Similarly, the storage control units 4a, 4b...4n can be separate physical units, or they can be virtual units, implemented, for example, using allocated resources from partitions of one or more servers.

[0035] Fig. Figure 2 shows in more detail an example of a data storage system 11 in which the restoration of the connection speed according to the present description is applied. The data storage system 11 is similar to the data storage systems of Fig. 1 and stands for those which include the memory control units 4a, 4b...4n and the memory 10a, 10b...10n. Fig. Figure 3 illustrates an example of a storage system comprising a primary data storage system 11a and a secondary data storage system 11b, in which link speed recovery according to an aspect of the present description is applied to one or both of the primary or secondary data storage systems.

[0036] Each data storage system 11 ( Fig. 2), 11a, 11b ( Fig. 3) includes a storage control unit or control unit 4 ( Fig. 2), 4a ( Fig. 3), 4b, which are based on data on data carriers 12 ( Fig. 2), Data carrier 1, Data carrier 2 ( Fig. 3) (e.g., logical units (LUNs), logical devices, logical subsystems, etc.) accessing the memory accessed by one or more memory units 10 ( Fig. 2), 10a, ( Fig. 3), 10b ( Fig. 3) is shown. Each memory control unit 4, 4a, 4b comprises a CPU complex 14 ( Fig. 2) comprising processor resources provided by one or more processors or central processing units, each containing one or more processor cores. In this embodiment, a processor core contains the components of a CPU involved in executing instructions, for example, an arithmetic logic unit (ALU), a floating-point unit (FPU), and various levels of cache (such as L1 and L2 cache). It is understood that a processor core may contain other logic elements besides, or in place of, those mentioned herein.

[0037] Each storage control unit 4 ( Fig. 2), 4a ( Fig. 3), 4b also contains a main memory 20 ( Fig. 2), which includes a memory manager 24 for managing memory operations in response to an I / O data request from a host, including writing data to or reading data from an associated memory 10, 10a ( Fig. 3) or 10b. The storage operations managed by the storage manager 24 also include data replication operations from a primary storage device1 ( Fig. 3) of a primary data storage system, such as data storage system 11a, for example, to a secondary data storage system 2, such as data storage system 11b. The storage manager 24 is configured to make copies of the primary data storage system 1 ( Fig. 3) of the primary data storage system 11a as a secondary data carrier2 ( Fig. 3) of the secondary data storage system 11b. The pair of disks, Disk1, Disk2, is in a copy relationship, so that updates to the first Disk1 are replicated to the second Disk2.

[0038] The CPU complex 14 of each memory control unit can contain multiple clusters of processors, each cluster containing its own allocated main memory 20, memory manager 24, cache, etc. The processing and main memory resources allocated to each cluster can be physical, virtual, shared, transferable, or dedicated, depending on the application.

[0039] As described in more detail below, a multi-loop logic 32 for restoring the connection speed of the memory manager 24 enables, in one embodiment, the successful training of a data transmission connection such as the connection 38 of the network 6b ( Fig. 1) which connects the memory control unit 4 to the memory 10. As a result, the connection 34, which forms part of a data transmission path 38 connecting the memory manager 4 to the memory 10, can be reliably trained or retrained to establish or restore full connection speed, thereby improving the efficiency of computer operations of the data storage system 11. Depending on the specific application, other aspects and advantages may be realized.

[0040] During the Fig. 2 and Fig. In the illustrated embodiment 4, the data transmission path 38 is a PCIe data transmission path comprising a data transmission path host 44, which may be implemented with a root complex. The data transmission path 38 ( Fig. 2, Fig. 4) further comprises a data transmission path switch 48, which contains a plurality of upstream input / output ports, one example of which is the upstream I / O port 52 ( Fig. 4) is shown, which includes an upstream optical transceiver 60. In one embodiment, the data transmission path switch 48 can be implemented as a removable add-on card containing a carrier for integrated circuit components of the data transmission path switch 48. Connectors arranged on the PCIe card of the data transmission path switch 48 enable a detachable connection of the data transmission path switch 48 to the data transmission path host 44 of the data transmission path 38.

[0041] The data transmission link 34 comprises one or more optical fiber cables that are optically connected to an optical transceiver 60 of the upstream connector 52. Although the link 34 is described as an optical fiber connection in the illustrated embodiment, it is understood that other data transmission technologies, such as cables made of conductive metals, wireless transmission, etc., can be used for the link 34. While the data transmission path 38 is depicted as providing a serial PCIe bus data transmission path, it is further understood that other data transmission protocols and technologies, such as other types of serial buses, parallel buses, etc., can be used for a data transmission path where the link speed restoration described herein is applied.

[0042] The data transmission path 38 further comprises an endpoint unit 64, which may contain a plurality of downstream ports, one example of which is shown as a downstream port 68 containing an optical transceiver 72. In this embodiment, the data transmission link 34 is connected at one end to the optical transceiver 60 of the upstream port 52 of the switch 48 and at another end to the optical transceiver 72 of the downstream port 68 of the endpoint unit 64 of a storage unit 10.

[0043] Although data transfer path 38 is depicted as providing a data transfer path between a storage control unit 4 and a memory 10, it is understood that a data transfer path where link speed restoration is applied as described herein can be used for data exchange between other types of units. For example, a data transfer path where link speed restoration is applied as described herein can provide data exchange between a host, such as host 1a, and, for example, a storage control unit 4. Other units using a data transfer path can also benefit from link speed restoration as described herein, depending on the application.

[0044] In the illustrated embodiment, the multi-loop logic 32 for restoring the connection speed of the memory control unit 4 is represented as software stored in main memory 20 and executed by the CPU complex 14. However, it is understood that the logic functions of the multi-loop logic 32 for restoring the connection speed can be implemented as hardware, software, firmware, or any combination thereof, depending on the application. For example, logic functions of the multi-loop logic 32 for restoring the connection speed can be implemented in a driver for a data transmission path such as, for example,for a PCIe data transfer path 38 and can also be implemented in addition to or instead of the implementation in the driver for the data transfer path in hardware, software, firmware or in any combination of one or more of these of the data transfer path 38.

[0045] In another aspect of restoring the link speed according to the present description, the data transmission path 38 comprises subordinate out-of-band paths (subordinate OOB paths (OOB = out-of-band)) 76a, 76b and an endpoint out-of-band control unit 80, which provide a subordinate data transmission path between the data transmission path host 44 and the endpoint unit 64. In one embodiment, the subordinate out-of-band paths (OOB paths) 76a, 76b can conform to the RS 485 serial protocol instead of the PCI protocol. Depending on the application, other protocols may be used. As described in more detail below, the subordinate out-of-band paths 76a, 76b and the out-of-band control unit 80 enable data transmission between the data transmission path host 44, such as a root complex, and the components of the endpoint unit 64, independently of the data transmission link 34.If, for example, the data transmission link 34 lacks operational data transmission lanes, data transmission can still be provided via the subordinate out-of-band paths 76a, 76b and the out-of-band control unit 80 to initiate a restoration of the link speed at the endpoint unit 64, as described in more detail below. In the illustrated embodiment, the data transmission path host 44 is connected to the out-of-band control unit 80 via the subordinate out-of-band path 76a, which in turn is connected to the endpoint unit 64 via the subordinate out-of-band path 76b to enable out-of-band data transmission between the data transmission path host 44 and the components of the endpoint unit 64.

[0046] In one embodiment, the memory or storage drives can be 10 ( Fig. 2), 10a, 10b...10n ( Fig. 1) In addition to a data transfer path endpoint unit 64, it may consist of one or more sequentially accessed storage units, such as hard disk drives and magnetic tape, or it may include non-sequentially accessed storage units, such as solid-state drives (SSDs). Each storage drive 10, 10a, 10b...10n may have a single sequentially or non-sequentially accessed storage unit, or it may have an array of storage units, such as a Just a Bunch of Disks (JBOD), a Direct Access Storage Device (DASD), a Redundant Array of Independent Disks (RAID), a virtualization unit, tape storage, flash memory, etc.

[0047] The storage units of storage drives 10, 10a, 10b...10n can be configured to store data in subordinate units of data storage, such as disks, tracks, memory areas, blocks, pages, segments, cylinders, etc. While the restoration of connection speed described herein refers to subordinate storage units such as disks, it is understood that the restoration of connection speed described herein is also applicable to other subordinate storage units such as tracks, memory areas, blocks, pages, segments, cylinders, etc.

[0048] System components 1a, 1b...1n, 4, 4a, 4b,...4n, 10, 10a, 10b...10n are connected to networks 6a, 6b, which enable data transmission between these components via switches, links, and endpoint units such as adapters. Thus, in one embodiment, networks 6a, 6b comprise a structure that can include a storage area network (SAN), a local area network (LAN), an intranet, the internet, a wide area network (WAN), a peer-to-peer network, a wireless network, an arbitrated loop network, etc. Data transfer paths from the storage systems to the hosts 1a, 1b...1n and from the storage control units 4, 4a, 4b...4n to the storage 10, 10a, 10b...10n can be based on various connection protocols such as Fibre Connection (FICON).Other data transmission paths within the structure can include, for example, a Fibre Channel arbitrated loop configuration, a serial loop architecture, or a bus interface such as a PCI interface (PCI = Peripheral Component Interconnect), for example, a PCI Express interface. The data transmission paths within the structure can also be part of an Ethernet network, so that each node has an individual network address (Internet Protocol address). Depending on the application, other types of data transmission paths can be used, such as a modem-to-telephone connection, a wireless network, etc.

[0049] Data transmission software associated with data transmission paths includes instructions and other software that control data transmission protocols and the operation of the data transmission hardware, as appropriate, according to the data transmission protocols. It is understood that different data transmission path protocols may be used depending on the application in question.

[0050] A typical host, which is provided by host 1a from Fig. 5 is represented, comprising a CPU complex 202 and a main memory 204, which contains an operating system 206 and an application 208, which work together to access data from the primary memory 10a via a memory control unit 4, 4a, 4b...4n ( Fig. 3) or to read from and write data updates to secondary storage 10b. An example of a suitable operating system is z / OS. It is understood that other types of operating systems may be used depending on the application.

[0051] The restoration of the connection speed according to the present description can be applied to any computer system that contains data transmission links by using in the logic provided by the multi-loop logic 32 for restoring the connection speed ( Fig. 2) is shown. Thus, to restore the connection speed, multi-loop logic can also be used for restoring the connection speed on each host, such as host 1a.

[0052] The hosts 1a, 1b...1n, the memory control units 4, 4a, 4b, the memory units 10, 10a, 10b, the data transmission path 38, and the multi-loop logic 32 for restoring link speed can each be implemented using any data processing unit modified to restore link speed as described herein. Data processing units suitable for modification as described herein include those currently known to the art, such as a personal computer, a workstation, a server, a mainframe computer, a handheld computer, a palmtop computer, a telephone unit, a network device, a blade computer, a processing unit, etc. For the hosts 1a, 1b...1n, the memory control units 4, 4a, 4b...4n, the memory units 10, 10a, 10b...10n, the data transmission path 38 and the multi-loop logic 32 for restoring connection speed can be elements in any network such as a storage network, a wide area network, the Internet, an intranet or elements in a cloud computing environment.

[0053] Fig. 6 represents an embodiment of operations of the multi-loop logic 32 for restoring the connection speed ( Fig. 2) the storage control unit 4 in connection with the data transmission path 38 ( Fig. 4) is shown. In this example, logic elements of the multi-loop logic 32 are used to restore the connection speed ( Fig. 2) and data transmission path 38 ( Fig. 4) configured to restore connection speed as described in Fig. 6 and as described in the attached description.

[0054] In one example, the link speed restoration operations are directed towards retraining the data transmission link 34 of the data transmission path 38 in order to regain a link speed, such as full link speed, after a loss of full link speed, in which one or more lanes of the data transmission link 34 were lost. As explained in detail below, link speed restoration in this embodiment involves repeating a main loop of successive link speed restoration commands a predetermined maximum number of times. Fig. Figure 7A shows an example of a set of consecutive commands for the main loop of consecutive commands to restore connection speed. It is understood that, depending on the application in question, the main loop may contain additional or alternative commands. Fig. Figure 7A illustrates that other sequences, types, or quantities of link speed restoration commands and operations may be used. It is further understood that the number of distinct main link speed restoration command loops may vary depending on the specific application. In this embodiment, each main link speed restoration command execution includes repeating a sub-loop of successive link speed restoration commands within each main loop execution a second predetermined maximum number of times.

[0055] Fig. Figure 7B shows an example of a set of sequential commands for the child loop of sequential link speed restoration commands. It is assumed that separating sequential link speed restoration commands into distinct sets can improve the reliability of link speed restoration. For example, it is assumed that by repeating the execution of a child loop of sequential link speed restoration commands within each main loop execution, and repeating the execution of the main loop of sequential link speed restoration commands according to one embodiment, the reliability of link speed restoration can be improved to full link speed.Thus, in one embodiment, the set of consecutive instructions for the child loop of link speed recovery commands is repeated more frequently than the set of consecutive instructions for the main loop of link speed recovery commands. Such an arrangement can improve the success rate with which retraining the link leads to the restoration of full link speed. For example, executing a set of consecutive instructions for the child loop of link speed recovery commands may be more successful than executing the set of consecutive instructions for the main loop.Therefore, repeating the set of consecutive commands for the sub-loop of consecutive commands to restore connection speed more frequently than repeating the set of consecutive commands for the main loop of consecutive commands to restore connection speed can improve the success rate of retraining the connection to full connection speed. It is understood that, depending on the application in question, a sub-loop may contain additional or alternative commands to the one in the main loop. Fig. Other sequences, types, or sets of commands and operations shown in Figure 7B may be used to restore connection speed. It is further understood that the number of different sub-loops of commands used to restore connection speed may vary depending on the application.

[0056] The multi-loop logic 32 for restoring connection speed ( Fig. 2) The memory control unit 4 starts in response to the loss of full connection speed (Block 220, Fig. 6) Restoring the connection speed to train data transmission link 34 of data transmission path 38. The main loop of successive commands to restore the connection speed is initially executed when restoring the connection speed of Fig. 6 started (Block 224, Fig. 6) and the next command to restore the link speed of the set of successive main loop commands ( Fig. 7A) is output (Block 226, Fig. 6) In this example, the next main loop instruction in the sequence ML1 to ML4 of main loop instructions is the first main loop instruction ML1 of the set of consecutive main loop instructions. As in Fig. 7A shows the next main loop instruction ML1 (block 228, Fig. 6) the subordinate loop of commands to restore the connection speed SL1 to SL4, which is in Fig. 7B are shown, and the next command to restore the connection speed of the set of successive commands of the sub-loop ( Fig. 7B) is output (Block 232, Fig. 6).

[0057] In this example, the next instruction of the child loop in the sequence SL1 to SL4 of child loop instructions is the first child loop instruction SL1 of the set of consecutive child loop instructions. As in Fig. As shown in 7B, the next command SL1 of the sub-loop causes the data transmission connection 34 ( Fig. 4) is retrained (or retrained) at the upstream port 52. In this embodiment, the data transmission path switch 48 includes an upstream link control unit 234, which controls the upstream port 52 in response to commands from the data transmission path host 44. Accordingly, the multi-loop logic 32 for restoring the link speed ( Fig. 2) the memory control unit 4 issues suitable commands to the data transfer path host 44, which in this example is a root complex, and the data transfer path host 44 issues suitable commands to the data transfer path switch 48 and its upstream port 52 to start retraining (or training) the data transfer link 34 connected to the upstream port 52.

[0058] In this embodiment, the retraining (or training) of the data link 34 can be initiated by setting a bit of register 236 of the upstream interface 52. For example, the PCIe specification provides a root complex to initiate the retraining of a link by setting a configuration bit of the PCI Express Capability Link Status Control register. Thus, in this embodiment, register 236 of the upstream interface 52 can be a PCI Express Capability Link Status Control register. It is understood that register 236 may have different formats depending on the specific protocol of the data link 34.

[0059] It is understood here that a single setting of a bit in register 236 of the upstream interface 52 may not be sufficient to successfully retrain the data transmission link 34. Accordingly, (Block 238, Fig. 6) the multi-loop logic 32 for restoring the connection speed ( Fig. 2) After setting the register bit of the memory control unit 4, which initiates the retraining of the data transmission link, and after a suitable delay to allow the retraining to complete, it checks whether the target connection speed, such as the full connection speed, has been reached by retraining (or training) the data transmission link initiated at the upstream port 52. In one embodiment, the connection speed of the data transmission links can be determined by querying the status of the data transmission link 34. When the full connection speed for the data transmission link 34 has been restored (or reached), the restoration of the connection speed is terminated (Block 240, Fig. 6).

[0060] Conversely, if it is determined that (Block 238, Fig. 6) that the desired full connection speed for the data transmission connection 34 was not achieved by retraining the connection speed, which was started at the upstream port 52 by setting the corresponding configuration bit of register 236, the multi-loop logic 32 determines to restore the connection speed ( Fig. 2) of the memory control unit 4 (block 244), whether all successive instructions to restore the connection speed of the set of successive instructions of the sub-loop ( Fig. 7B) were carried out.

[0061] In this example, not all consecutive commands to restore the connection speed of the set of consecutive commands of the child loop were executed. Fig. 7B). Accordingly, the next command to restore the connection speed of the set of successive commands of the sub-loop ( Fig. 7B) output (Block 232, Fig. 6) In this example, the next instruction of the child loop in the sequence SL1 to SL4 of child loop instructions is the second child loop instruction SL2 of the set of consecutive child loop instructions. As in Fig. As shown in 7B, the next instruction SL2 of the sub-loop causes the data transmission connection 34 ( Fig. 4) is retrained (or trained) at the downstream port 68. In this embodiment, the data transmission path endpoint 64 comprises a downstream link control unit 246, which controls the downstream port 68 in response to commands from the data transmission path host 44.

[0062] In one aspect of restoring the link speed according to the present description, the subordinate out-of-band path 76a provides a subordinate data transmission path between the data transmission path host (root complex) 44 and the endpoint out-of-band control unit 80, which in turn is connected to the data transmission path endpoint unit 64 via the subordinate out-of-band data transmission path 76b. The subordinate out-of-band paths 76a, 76b, and the control unit 80 enable data exchange between the data transmission path host 44 and the downstream link control unit 246 of the endpoint unit 64, independent of the data exchange link 34.For example, if, due to a failure of the retraining (or training) described above at the upstream interface to establish or restore the data transmission lanes for data link 34, data link 34 has lost all data transmission lanes (or has not established any), data transmission between the data link host 44 and the downstream link control unit 246 can still be provided by the subordinate out-of-band paths 76a and 76b and the endpoint out-of-band control unit 80. Accordingly, the multi-loop logic 32 for restoring the link speed ( Fig. 2) the memory control unit 4 issues suitable commands to the data transmission path host 44, which in this example is a root complex, and in response the data transmission path host 44 issues suitable commands to the data transmission path endpoint 64 and its downstream port 68 via the endpoint out-of-band control unit 80 and the subordinate out-of-band paths 76a, 76b to start retraining the data transmission link 34 connected to the downstream port 68.

[0063] In this embodiment, retraining or training of the data link 34 can also be initiated by setting a bit of register 248 of the downstream port 68. As noted above, the PCIe specification provides a root complex to initiate the retraining of a link by setting a configuration bit of the PCI Express Capability Link Status Control register. Thus, in this embodiment, register 248 of the downstream port 68 can be a PCI Express Capability Link Status Control register. It is understood that register 248 may have different formats depending on the specific protocol of the data link 34.

[0064] It is understood here that by setting a single bit of register 248 of the downstream port 68, successful retraining or training of the data transmission link 34 may not be achieved. Accordingly, (Block 238, Fig. 6) the multi-loop logic 32 for restoring the connection speed ( Fig. 2) the memory control unit 4 after setting the register bit, thereby initiating the retraining of the data transmission link at the downstream port, and after a suitable delay to allow the retraining to complete, to determine whether the desired link speed, such as the full link speed, has been achieved by retraining the data transmission link initiated at the downstream port 68. In one embodiment, the link speed of the data transmission links can be determined by querying the status of the data transmission link 34. When the full link speed for the data transmission link 34 has been restored, the link speed restoration is terminated (Block 240, Fig. 6).

[0065] Conversely, if it is determined that (Block 238, Fig. 6) that the desired full connection speed for the data transmission connection 34 was not achieved by retraining the connection speed, which was started at the downstream port 68 by setting the corresponding configuration bit of register 248, the multi-loop logic 32 determines to restore the connection speed ( Fig. 2) of the memory control unit 4 (block 244) again, whether all successive instructions to restore the connection speed of the set of successive instructions of the sub-loop ( Fig. 7B) were carried out.

[0066] In this example, not all consecutive commands to restore the connection speed of the set of consecutive commands of the child loop were executed. Fig. 7B). Accordingly, the next command to restore the connection speed of the set of successive commands of the sub-loop ( Fig. 7B) output (Block 232, Fig. 6) In this example, the next instruction of the child loop in the sequence SL1 to SL4 of child loop instructions is the third child loop instruction SL3 of the set of consecutive child loop instructions. As in Fig. As shown in 7B, the next command SL3 of the sub-loop causes the data transmission connection 34 ( Fig. 4) is retrained or retrained at the upstream port 52 by starting a disable-enable cycle at the upstream port 52. Accordingly, the multi-loop logic 32 for restoring the connection speed ( Fig. 2) The storage control unit 4 issues suitable commands to the data transmission path host 44, which in this example is a root complex, and in response, the data transmission path host 44 issues suitable commands to the data transmission path switch 48 and its upstream port 52 to initiate a disable-enable cycle at the upstream port 52 to start retraining or training the data transmission link 34 connected to the upstream port 52. In this embodiment, training or retraining the data transmission link 34 can be initiated by disabling the upstream port, waiting for a suitable duration, and then enabling the upstream port in a disable-enable cycle. For example, a wait in the range of 50 microseconds to one millisecond, such as100 microseconds may be appropriate, depending on the application in question.

[0067] It is understood here that a single deactivation-activation-switching of the upstream port 52 may not be sufficient to successfully train or retrain the data transmission link 34. Accordingly, it was determined (Block 238, Fig. 6) the multi-loop logic 32 for restoring the connection speed ( Fig. 2) After the disable-enable cycle and a suitable delay to allow the retraining initiated by the disable-enable cycle to complete, the memory control unit 4 checks whether the desired link speed, such as full link speed, has been restored by retraining the data link initiated at the upstream port 52. If full link speed has been restored for data link 34, the link speed restoration process is terminated (Block 240, Fig. 6).

[0068] Conversely, if it is determined that (Block 238, Fig. 6) that the full connection speed for the data transmission connection 34 was not achieved by retraining the connection speed, which was started at the upstream port 52 by deactivating-activating-switching the upstream port 52, the multi-loop logic 32 determines to restore the connection speed ( Fig. 2) the memory control unit 4 (block 244) again, whether all successive instructions to restore the connection speed of the set of successive instructions of the sub-loop ( Fig. 7B) were carried out.

[0069] In this example, not all consecutive commands to restore the connection speed of the set of consecutive commands of the child loop were executed. Fig. 7B). Accordingly, the next command to restore the connection speed of the set of successive commands of the sub-loop ( Fig. 7B) output (Block 232, Fig. 6) In this example, the next instruction of the child loop in the sequence SL1 to SL4 of child loop instructions is the fourth child loop instruction SL4 of the set of consecutive child loop instructions. As in Fig. As shown in 7B, the next instruction SL4 of the sub-loop causes the data transmission connection 34 ( Fig. 4) is retrained or retrained at the downstream port 68 by starting a disable-enable cycle at the downstream port 68. Accordingly, the multi-loop logic 32 for restoring the link speed ( Fig. 2) the storage control unit 4 issues suitable commands to the data transmission path host 44, which in this example is a root complex, and in response the data transmission path host 44 issues suitable commands to the data transmission path endpoint 64 and its downstream port 68 via the subordinate out-of-band path 76a, the endpoint out-of-band control unit 80 and the subordinate out-of-band path 76b to initiate a disable-enable cycle at the downstream port 68 to start retraining the data transmission link 34 connected to the downstream port 68.

[0070] In this embodiment, the training or retraining of the data transmission link 34 can be initiated by disabling the downstream port, waiting a suitable duration, and then activating the downstream port in a disable-activate cycle. For example, a wait in the range of 50 microseconds to one millisecond, such as 100 microseconds, may be appropriate depending on the application. It is understood here that a single disable-activate toggle of the downstream port 68 may not be sufficient to successfully train or retrain the data transmission link 34. Accordingly, the multi-loop logic 32 for restoring the link speed follows ( Fig. 2) of the memory control unit 4 after the start of the disable-enable cycle and after a suitable delay to allow the execution of the training or retraining started by the disable-enable cycle, the disable-enable cycle with a determination (block 238, Fig. 6) whether the desired connection speed, such as full connection speed, has been achieved by training or retraining the data link, which was started at the downstream port 68. If full connection speed has been restored for data link 34, the connection speed restoration process is terminated (Block 240, Fig. 6).

[0071] Conversely, if it is determined that (Block 238, Fig. 6) that the full connection speed for the data transmission connection 34 was not achieved by retraining the connection speed, which was started at the downstream port 68 by disabling-enabling-switching the downstream port 68, the multi-loop logic 32 determines to restore the connection speed ( Fig. 2) the memory control unit 4 (block 244) again, whether all successive instructions to restore the connection speed of the set of successive instructions of the sub-loop ( Fig. 7B) were carried out.

[0072] In this example, all consecutive instructions SL1 to SL4 were used to restore the connection speed of the set of consecutive instructions in the sub-loop ( Fig. 7B) as described above. Accordingly determined (Block 250, Fig. 6) the multi-loop logic 32 for restoring the connection speed ( Fig. 2) the memory control unit 4, whether a maximum execution of the sub-loop has been reached. In one embodiment, the execution of the sub-loop of successive instructions SL1 to SL4 is repeated within each main loop to restore the link speed up to a maximum represented by a variable “S”, which may be, for example, “5”. It is understood that other maxima may be selected depending on the application in question, for example, in a range of 5 to 10. If it is determined (block 250, Fig. 6) that the maximum S was not reached by repeating the execution of the sub-loop of consecutive commands SL1 to SL4 to restore the link speed within the main loop, the execution of the sub-loop of consecutive commands SL1 to SL4 to restore the link speed is repeated at blocks 228 to 244 as described above until the maximum S is reached. Once it is determined (block 250, Fig. 6) that the maximum S within the main loop has been reached by repeating the subordinate loop of successive commands SL1 to SL4 to restore the link speed, the execution of the main loop of successive commands ML1 to ML4 to restore the link speed (blocks 252A, 252B) is continued.

[0073] Accordingly, the next command to restore the connection speed of the set of consecutive main loop commands ( Fig. 7A) output (Block 256, Fig. 6) In this example, the next main loop instruction in the sequence ML1 to ML4 of main loop instructions is the second main loop instruction ML2 of the set of consecutive main loop instructions. As in Fig. As shown in 7A, the next main loop instruction ML2 causes the data transfer path switch 48 to be reset and then its configuration to be reinitialized.

[0074] Accordingly, the multi-loop logic 32 provides instructions for restoring the connection speed ( Fig. 2) the storage control unit 4 issues suitable commands to the data transmission path host 44, which in this example is a root complex, and in response the data transmission path host 44 issues suitable commands to the data transmission path switch 48 to reset it and then reinitialize its configuration to start retraining or training the data transmission link 34, which is connected to the upstream port 52.

[0075] In one embodiment, resetting the data transmission path switch 48 is a fundamental reset, which restores all configuration bits of the configuration registers of the data transmission path switch 48 to their default values. Accordingly, the reset bits are reinitialized after a fundamental reset, if necessary. It is understood that other types of resets can be performed, such as a hard reset or a soft reset, in which different levels of register bits are reset and then reinitialized.

[0076] In response to resetting and subsequently reinitializing the data transmission path switch, it is known that training of data transmission link 34 is started automatically. However, one aspect of restoring the connection speed according to the present description is that resetting and subsequently reinitializing the data transmission path switch 48 may not result in successful training or retraining of data transmission link 34. Accordingly, it was determined (Block 260, Fig. 6) the multi-loop logic 32 for restoring the connection speed ( Fig. 2) After resetting and subsequently reinitializing the data transmission path switch 48 and after a suitable delay, the storage control unit 4 checks whether the target connection speed, such as the full connection speed, has been achieved by retraining or training the data transmission link that was started at the upstream port 52. In one embodiment, the connection speed of the data transmission links can be determined by querying the status of the data transmission link 34. When the full connection speed for the data transmission link 34 has been restored, the restoration of the connection speed is terminated (Block 240, Fig. 6).

[0077] Conversely, if it is determined (Block 260, Fig. 6) that the full connection speed for the data transmission link 34 was not achieved by training or retraining the connection speed, which was started at the upstream port 52 by resetting and then reinitializing the data transmission path switch 48, the multi-loop logic 32 determines to restore the connection speed ( Fig. 2) of the memory control unit 4 (block 264), whether all successive instructions to restore the link speed of the set of successive main loop instructions ( Fig. 7A) were carried out.

[0078] In this example, not all consecutive commands to restore the link speed of the set of consecutive main loop commands were executed ( Fig. 7A). Accordingly, the next instruction to restore the link speed of the set of consecutive main loop instructions ( Fig. 7A) output (Block 256, Fig. 6) In this example, the next main loop instruction in the sequence ML1 to ML4 of main loop instructions is the third main loop instruction ML3 of the set of consecutive main loop instructions. As in Fig. As shown in Figure 7A, the next main loop instruction ML3 causes the optical transceiver 60 of the upstream port 52 to be reset and restarted.

[0079] Accordingly, the multi-loop logic 32 provides instructions for restoring the connection speed ( Fig. 2) the storage control unit 4 issues suitable commands to the data transmission path host 44, which in this example is a root complex, and in response the data transmission path host 44 issues suitable commands to the data transmission path switch 48 to reset the optical transceiver 60 of the upstream port 52 in order to start retraining the data transmission link 34 which is connected to the upstream port 52.

[0080] In response to the resetting of the optical transceiver 60 of the upstream port 52, it is known that the training of the data transmission link 34 is started automatically. However, with regard to one aspect of restoring the link speed according to the present description, it is apparent that resetting the optical transceiver 60 of the upstream port 52 may not result in successful training or retraining of the data transmission link 34. Accordingly, it was determined (Block 260, Fig. 6) the multi-loop logic 32 for restoring the connection speed ( Fig. 2) After resetting the optical transceiver 60 of the upstream port 52 and waiting a sufficient time while the optical transceiver 60 restarts and retrains, the storage control unit 4 checks whether the desired connection speed, such as the full connection speed, has been achieved by retraining the data transmission link, which was initiated at the upstream port 52 by resetting the optical transceiver 60 of the upstream port 52. In one embodiment, a waiting period of approximately 5 seconds may be appropriate, depending on the application. As noted above, the connection speed of the data transmission links can be determined by querying the status of the data transmission link 34.When full connection speed has been restored for data transmission link 34, the connection speed restoration process will end (Block 240, . Fig. 6).

[0081] Conversely, if it is determined (Block 260, Fig. 6) that the full link speed for the data transmission link 34 was not achieved by training or retraining the link speed, which was started by resetting the optical transceiver 60 of the upstream port 52, determines (Block 264) the multi-loop logic 32 to restore the link speed ( Fig. 2) of the memory control unit 4, whether all successive instructions to restore the link speed of the set of successive main loop instructions ( Fig. 7A) were carried out.

[0082] In this example, not all consecutive commands to restore the link speed of the set of consecutive main loop commands were executed ( Fig. 7A). Accordingly, the next instruction to restore the link speed of the set of consecutive main loop instructions ( Fig. 7A) output (Block 256, Fig. 6) In this example, the next main loop instruction in the sequence ML1 to ML4 of main loop instructions is the fourth main loop instruction, ML4, of the set of consecutive main loop instructions. As in Fig. As shown in Figure 7A, the next main loop instruction ML4 causes the downstream optical transceiver 72 of the downstream port 68 to be reset.

[0083] Accordingly, the multi-loop logic 32 provides instructions for restoring the connection speed ( Fig. 2) the storage control unit 4 issues suitable commands to the data transmission path host 44, which in this example is a root complex, and in response the data transmission path host 44 issues suitable commands to the downstream optical transceiver 72 of the downstream port 68 via the subordinate out-of-band path 76a, the endpoint out-of-band control unit 80 and the subordinate out-of-band path 76b to initiate a reset and restart of the optical transceiver 72 at the downstream port 68 in order to initiate the training or retraining of the data transmission link 34 connected to the downstream port 68.

[0084] In response to the resetting of the optical transceiver 72 of the downstream port 68, it is known that the training of the data transmission link 34 is started automatically. However, with regard to one aspect of restoring the link speed according to the present description, it is apparent that a single resetting of the optical transceiver 72 of the downstream port 68 may not result in successful training or retraining of the data transmission link 34. Accordingly, it was determined (Block 260, Fig. 6) the multi-loop logic 32 for restoring the connection speed ( Fig. 2) After resetting the optical transceiver 72 of the downstream port 68 and waiting a sufficient time while the optical transceiver 72 restarts and trains, the storage control unit 4 checks whether the desired link speed, such as full link speed, has been achieved by training or retraining the data transmission link, which was initiated at the downstream port 68 by resetting the optical transceiver 72 of the downstream port 68. In one embodiment, depending on the application, a waiting period of approximately 5 seconds may be appropriate. When full link speed for the data transmission link 34 has been restored, the link speed restoration process is terminated (Block 240, Fig. 6).

[0085] Conversely, if it is determined (Block 260, Fig. 6) that the full link speed for the data transmission link 34 was not achieved by training or retraining the link speed, which was started by resetting the optical transceiver 72 of the downstream port 68, determines (block 264) the multi-loop logic 32 to restore the link speed ( Fig. 2) of the memory control unit 4, whether all successive instructions to restore the link speed of the set of successive main loop instructions ( Fig. 7A). In this example, all consecutive instructions ML1 to ML4 were executed to restore the link speed of the set of consecutive main loop instructions ( Fig. 7A) as described above. Accordingly determined (Block 268, Fig. 6) the multi-loop logic 32 for restoring the connection speed ( Fig. 2) the memory control unit 4, whether a maximum execution of the main loop has been reached. In one embodiment, the execution of the main loop of successive instructions ML1 to ML4 is repeated to restore the connection speed up to a maximum represented by a variable “M”, which may be, for example, “3”. However, it is understood that other maxima may be selected depending on the application in question, for example, in a range of 3 to 5. If it is determined (Block 268, Fig. 6) If the repetition of the main loop execution of successive commands ML1 to ML4 to restore the link speed has not reached the maximum M, the main loop execution of successive commands ML1 to ML4 to restore the link speed is repeated at blocks 224 to 264 as described above until the maximum M is reached.

[0086] It is assumed that training or retraining the data link 34 to full link speed is frequently achieved before the maximum M of main loop repetitions is reached. For example, it is assumed that by repeating the execution of a sub-loop of successive link speed recovery commands within each main loop execution, and repeating the execution of the main loop of successive link speed recovery commands according to one embodiment, the reliability of restoring the link speed to full link speed can be improved. However, if it is found (Block 268, Fig. 6) that the repetition of the main loop execution of successive commands ML1 to ML4 to restore the link speed has reached the maximum M without the retraining of the data transmission link 34 to the full link speed, in one embodiment the main loop execution of successive commands ML1 to ML4 to restore the link speed can be terminated (block 240). In addition, any faulty or defective components of the data transmission path can be replaced and the process of Fig. Step 6 must be repeated to restore connection speed.

[0087] The calculation components of the figures can each be implemented in one or more computer systems, such as in the one in Fig. The computer system / server 1002 shown in Figure 8 can be implemented. The computer system / server 1002 can be described in the general context of instructions executable on a computer system, such as program modules that are executed by a computer system. Program modules can generally contain routines, programs, objects, components, logic, and data structures, etc., that perform specific tasks or implement certain abstract data types. The computer system / server 1002 can be practically implemented in distributed cloud computing environments, where tasks are performed by remotely located processing units connected via a data transmission network. In a distributed cloud computing environment, program modules can reside on both local and remotely located computer system storage media, including main memory units.

[0088] As in Fig. Figure 8 shows the computer / server 1002 in the form of a universal data processing unit. The components of the computer system / server 1002 can include, but are not limited to, one or more processors or processing units 1004, a system main memory 1006, and a bus 1008 that connects various system components, including the system main memory 1006, to a processor 1004. The bus 1008 represents one or more different types of bus structures, including a memory bus or memory control unit, a peripheral bus, an accelerated graphics port, and a processor bus or local bus, utilizing any variety of bus architectures.Examples of such architectures, and not intended as a limitation, include an ISA bus (ISA = Industry Standard Architecture), an MCA bus (MCA = Micro Channel Architecture), the EISA bus (EISA = Enhanced ISA), a local VESA bus (VESA = Video Electronics Standards Association), and a PCI bus (PCI = Peripheral Component Interconnects).

[0089] The computer system / server 1002 typically contains a variety of media that can be read by computer systems. Such media can be any available media that the computer system / server 1002 can access, and this media includes both volatile and non-volatile media, removable media and non-removable media.

[0090] The system main memory 1006 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 1010 and / or cache 1012. The computer system / server 1002 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 1013 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown and usually referred to as a "hard disk"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium.In such cases, each of these units can be connected to the bus 1008 via one or more data media interfaces. As further illustrated and described below, the main memory 1006 can contain at least one program product, which contains a group (e.g., at least one) of program modules configured to execute the functions of embodiments of the invention.

[0091] A program / utility 1014 containing a group (at least one) of program modules 1016 can, by way of example and not as a limitation, be stored in main memory 1006 as well as in an operating system, in one or more application programs, or in other program modules and program data. Each of the operating systems, one or more application programs, other program modules and program data, or a specific combination thereof, can contain a form of implementation of a network environment. The components of the computer system 1002 can be implemented as program modules 1016, which generally perform the functions and / or methodologies of embodiments of the invention described herein. The system of Fig. 1 can be implemented in one or more computer systems 1002, whereby when implemented in multiple computer systems 1002 the computer systems can exchange data via a network.

[0092] The computer system / server 1002 can also exchange data with one or more external units 1018, for example, a keyboard, a pointing device, a display unit 1020, etc.; with one or more units that allow a user to interact with the computer system / server 1002; and / or with any units (e.g., network card, modem, etc.) that allow the computer system / server 1002 to exchange data with one or more other data processing units. Such data exchange can take place via input / output (I / O) interfaces 1022. Furthermore, the computer system / server 1002 can exchange data with one or more networks via a network adapter 1024, for example, a local area network (LAN), a wide area network (WAN), and / or a public network (e.g., the Internet).As shown, network adapter 1024 exchanges data with the other components of computer system / server 1002 via bus 1008. It should be clear that, although not shown, other hardware and / or software components could be used in conjunction with computer system / server 1002. Examples include, but are not limited to: microcode, unit drivers, redundant processing units, external hard disk drive arrays, RAID systems (RAID = Redundant Array of Independent Disks), tape drives, storage systems for data archiving, etc.

[0093] The reference symbols used herein, such as i, j and n, serve to denote a variable number of instances of an element, which can stand for the same value or different values, and can stand for the same or a different value when they are described in different described instances with the same element or with different elements.

[0094] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium (or media) on which computer-readable program instructions are stored to cause a processor to perform processor operations according to aspects of the present invention.

[0095] The computer-readable storage medium can be a physical unit on which instructions for use by a unit for executing instructions can be stored and retained. The computer-readable storage medium can be, for example, but is not limited to, an electronic storage unit, a magnetic storage unit, an optical storage unit, an electromagnetic storage unit, a semiconductor storage unit, or any suitable combination thereof.A non-exhaustive list of more precise examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable read-only memory in the form of a compact disc (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically coded unit such as punched cards or raised structures in a groove with instructions recorded on them, or any suitable combination of the foregoing.A computer-readable storage medium, as used herein, is not to be interpreted as consisting of volatile signals per se, such as radio waves or freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through an optical fiber cable), or electrical signals transmitted via a cable.

[0096] The computer-readable program instructions described herein can be downloaded over a network, such as the internet, a local area network, a wide area network, and / or a wireless network, from a computer-readable storage medium to relevant data processing units or to an external computer or storage device. The network may include copper transmission cables, fiber optic transmission lines, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface at each data processing unit receives computer-readable program instructions from the network and forwards them for storage on a computer-readable storage medium within the respective data processing unit.

[0097] Computer-readable program instructions for performing operations of the present invention may be assembly instructions, ISA instructions (ISA = Instruction Set Architecture), machine instructions, machine-dependent instructions, microcode, firmware instructions, data for setting states, or either source code or object code written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++ or similar, and conventional procedural programming languages ​​such as the programming language "C" or similar programming languages.The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be to an external computer (for example, via the internet using an internet service provider).In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), which execute computer-readable program instructions, can be used to personalize the electronic circuit by utilizing state information from the computer-readable program instructions, thus implementing aspects of the present invention.

[0098] Aspects of the present invention are described herein with reference to flowchart representations and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the invention. It will be clear that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented using computer-readable program instructions.

[0099] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a specialized computer, or other programmable data processing devices to create a machine such that the instructions executed by the processor of the computer or other programmable data processing devices provide the means to perform the functions / actions specified in a block or blocks of the flowchart(s) and / or block diagram(s).These computer-readable program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing devices, or other units to function in a certain manner, such that the computer-readable medium with instructions stored on it constitutes a product containing instructions that implement the function / action specified in a block or blocks of the flowcharts and / or block diagrams.

[0100] The computer-readable program instructions can also be loaded into a computer, other programmable data processing devices, or other units to cause a series of operations to be performed on the computer, other programmable devices, or other units to create a computer-realized process, such that the instructions executed on the computer, other programmable devices, or units realize the functions / actions specified in a block or blocks of the flowcharts and / or block diagrams.

[0101] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this context, each block in the flowcharts or block diagrams can represent a module, segment, or section of instructions that includes one or more executable instructions for implementing the specified logic function(s). In some alternative implementations, the functions specified in the block may be executed in a different order than that shown in the figures. For example, two consecutively listed blocks may actually be executed essentially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality associated with them.Furthermore, it should be noted that each block of the block diagrams and / or flowchart representations, as well as combinations of blocks in the block diagrams and / or flowchart representations, can be implemented using dedicated hardware-based systems to perform the specified functions or actions, or using combinations of dedicated hardware and dedicated computer instructions.

[0102] The terms “an embodiment”, “elaboration”, “elaborations”, “the embodiment”, “the embodiments”, “one or more embodiments”, “some embodiments” and “one embodiment” mean “one or more (but not all) embodiments of the present invention(s)”, unless expressly stated otherwise.

[0103] The terms “among others”, “include”, “have” and variations thereof mean “including, but not limited to”, unless expressly stated otherwise.

[0104] All numbered lists of elements do not imply that any or all of the elements are mutually exclusive, unless expressly stated otherwise.

[0105] The terms “ein / eine” and “der / die / das” refer to “one or more”, unless explicitly stated otherwise.

[0106] Units that are interconnected need not be in continuous contact unless explicitly stated otherwise. Furthermore, interconnected units can exchange data directly or indirectly via one or more intermediary units.

[0107] A description of an embodiment in which several components are interconnected does not implicitly imply that all of these components are required. Rather, a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0108] Where a single unit or item is described herein, it is readily apparent that more than one unit / item (regardless of whether they work together) may be used instead of a single unit / item. Likewise, where more than one unit or item is described herein (regardless of whether they work together), it is readily apparent that a single unit or item, or any other number of units / items, may be used instead of the number of units or programs shown. The functionality and / or features of a unit may alternatively be embodied by one or more other units not expressly described as containing such functionality / features. Therefore, other embodiments of the present invention need not include the unit itself.

[0109] The foregoing description of various embodiments of the invention is provided for illustrative and descriptive purposes. It is not intended to be exhaustive or to limit the invention in its precisely disclosed form. Many modifications and variations are possible in light of the teachings presented above. It is not intended that this detailed description limit the scope of protection of the invention, but rather that the scope is limited by the claims included herein. The above detailed description, examples, and data constitute a complete description of the manufacture and use of the invention's structure. Since many embodiments of the invention can be created without deviating from the core concept and scope of protection of the invention, the invention is set forth in the claims included below.

Claims

[1] Method which features: Performing a link speed restoration to achieve a target link speed on a data transmission link, comprising repeating a main loop of successive link speed restoration commands with a first predetermined maximum number of times, wherein each main loop execution of link speed restoration commands comprises repeating a sub-loop of successive link speed restoration commands within each main loop execution with a second predetermined maximum number of times, wherein each execution of the sub-loop of successive link speed restoration commands comprises: Issuing the next subsequent link speed restoration command of a first set of successive link speed restoration commands; Determine whether the target connection speed has been reached after issuing a subsequent command to restore the connection speed; and Terminate the link speed restoration process when the target link speed has been reached in response to a subsequent link speed restoration command; wherein each main loop execution of link speed restoration commands includes repeating the execution of the sub-loop of link speed restoration commands a second predetermined maximum number of times: Issuing a subsequent command to restore link speed, followed by a second set of consecutive commands to restore link speed; Determine whether the target connection speed has been reached after issuing a subsequent command to restore the connection speed; and Terminate the link speed restoration process when the target link speed has been reached in response to a subsequent link speed restoration command. [2] Method according to claim 1, wherein each main loop execution of commands to restore the connection speed further comprises, after repeating the execution of the main loop of commands to restore the connection speed with the first predetermined maximum number of times, terminating the execution of the connection speed restoration if, in response to a command to restore the connection speed, the target connection speed has not been reached. [3] Method according to claim 1, wherein each execution of the subordinate loop of commands to restore the link speed further comprises: Issuing another subsequent link speed restoration command of the first set of link speed restoration commands if the target link speed was not reached in response to a link speed restoration command of the first set; Determine whether the target connection speed has been reached after issuing a subsequent command to restore the connection speed of the first sentence; and Terminate the link speed restoration process when the target link speed has been reached in response to a subsequent link speed restoration command from the first set. [4] Method according to claim 1, wherein each main loop execution of commands to restore link speed further comprises: Issuing another subsequent link speed restoration command of the second set of link speed restoration commands if the target link speed was not reached in response to a link speed restoration command of the second set; Determine whether the target connection speed has been reached after issuing a subsequent command to restore the connection speed of the second sentence; and Terminate the link speed restoration process when the target link speed has been reached in response to a subsequent link speed restoration command of the second set. [5] Method according to claim 1, wherein the first set of successive link speed recovery instructions of the subordinate loop of successive link speed recovery instructions comprises at least one of setting a configuration bit of a register on an upstream port connected to the data transmission link, sending an out-of-band signal to set a configuration bit of a register of a downstream port connected to the data transmission link, starting a disable-enable cycle of the downstream port and sending an out-of-band signal to start a disable-enable cycle of the downstream port. [6] Method according to claim 1, wherein the second set of successive link speed recovery commands of the main loop of successive link speed recovery commands comprises at least one of initiating the subordinate link speed recovery loop, initiating a reset and reinitialization of a data transmission path switch comprising an upstream port containing an upstream optical transceiver connected to the data transmission link, resetting the upstream optical transceiver and sending an out-of-band signal to reset a downstream optical transceiver connected to the data transmission link. [7] System comprising means designed to perform all steps of the method according to any of the preceding claims. [8] A computer program comprising instructions for performing all steps of the method according to any of the preceding method claims when the computer program is executed on a computer system. [9] Computer program product configured for use with a computer system comprising a host and a data storage system comprising a storage control unit and at least one storage unit controlled by the storage control unit and configured to store data, wherein the computer system comprises a data transmission path switch comprising an upstream port with an upstream transceiver, an endpoint unit comprising a downstream port with a downstream transceiver, a data transmission link connected to the upstream port and the downstream port, and at least one processor, and wherein the computer program product comprises a computer-readable storage medium containing program instructions embodied therein, wherein the program instructions are executable by a processor of the computer system to perform computer system operations,the computer system operations include: Performing a link speed restoration to achieve a target link speed on the data transmission link, which includes repeating a main loop of successive link speed restoration commands with a first predetermined maximum number of times, wherein each main loop execution of link speed restoration commands includes repeating a sub-loop of successive link speed restoration commands within each main loop execution with a second predetermined maximum number of times, wherein each execution of the sub-loop of successive link speed restoration commands includes: Issuing the next subsequent link speed restoration command of a first set of successive link speed restoration commands; Determine whether the target connection speed has been reached after issuing a subsequent command to restore the connection speed; and Completion of the connection speed restoration process, when the target connection speed has been reached in response to a subsequent command to restore the connection speed; wherein each main loop execution of link speed restoration commands includes repeating the execution of the sub-loop of link speed restoration commands a second predetermined maximum number of times: Issuing a subsequent command to restore link speed, followed by a second set of consecutive commands to restore link speed; Determine whether the target connection speed has been reached after issuing a subsequent command to restore the connection speed; and Terminate the link speed restoration process when the target link speed has been reached in response to a subsequent link speed restoration command. [10] Computer program product according to claim 9, wherein the upstream transceiver is an optical transceiver and the downstream connector is an optical transceiver, and wherein the second set of successive link speed recovery commands of the main loop of successive link speed recovery commands comprises at least one of initiating the sub-loop of successive link speed recovery commands, initiating a reset and reinitialization of the data transmission path switch, resetting the upstream optical transceiver, and sending an out-of-band signal to reset the downstream optical transceiver. [11] Computer system for use with a host, comprising: a storage control unit; at least one storage unit that is controlled by the storage control unit and configured to store data; a data transmission path switch that has an upstream port with an upstream transceiver; an endpoint unit that has a downstream connection with a downstream transceiver; a data transmission link that is connected to the upstream port and the downstream port and is configured to transmit data between the upstream port and the downstream port; at least one processor in the computer system; and A computer program product comprising a computer-readable storage medium containing program instructions, wherein the program instructions are executable by a processor of the computer system to perform computer system operations, the computer system operations comprising: Performing a link speed restoration to achieve a target link speed on the data transmission link, which includes repeating a main loop of successive link speed restoration commands with a first predetermined maximum number of times, wherein each main loop execution of link speed restoration commands includes repeating a sub-loop of successive link speed restoration commands within each main loop execution with a second predetermined maximum number of times, wherein each execution of the sub-loop of successive link speed restoration commands includes: Issuing the next subsequent link speed restoration command of a first set of successive link speed restoration commands; Determine whether the target connection speed has been reached after issuing a subsequent command to restore the connection speed; and Completion of the connection speed restoration process, when the target connection speed has been reached in response to a subsequent command to restore the connection speed; wherein each main loop execution of link speed restoration commands includes repeating the execution of the sub-loop of link speed restoration commands a second predetermined maximum number of times: Issuing a subsequent command to restore link speed, followed by a second set of consecutive commands to restore link speed; Determine whether the target connection speed has been reached after issuing a subsequent command to restore the connection speed; and Terminate the link speed restoration process when the target link speed has been reached in response to a subsequent link speed restoration command.

Images