Systems and methods for input / output throttling in networks

By dynamically managing the suspend list buffer and queue buffer, the problems of uneven resource allocation and congestion in network communication systems are solved, achieving efficient IO throttling control and improving network throughput and system performance.

CN117914811BActive Publication Date: 2026-11-10AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202311317870.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2022-10-18
Filing Date
2023-10-12
Publication Date
2026-11-10
Estimated Expiration
2043-10-12

AI Technical Summary

Technical Problem

Existing network communication systems are prone to congestion during data transmission, leading to reduced network throughput. This is especially true when network nodes or links are overloaded beyond their processing capacity, making it difficult to effectively manage resource allocation and queue depth, thus impacting system performance.

Method used

A circuit system combining a suspend list buffer and a queue buffer is used to dynamically manage the queue depth value of each device. Unprocessed communications are stored in the suspend list, and communications are sent out when the queue depth allows, thereby achieving device-level I/O throttling control.

Benefits of technology

It effectively prevents resource depletion, ensures even distribution of communication resources, increases network throughput, reduces communication latency, improves system performance, and requires no operating system software intervention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117914811B_ABST
    Figure CN117914811B_ABST
Patent Text Reader

Abstract

This disclosure relates to systems and methods for input / output throttling in a network. Some embodiments relate to a method of communication in a network. The system and method are capable of providing a first communication associated with a device, issuing the first communication when the device's queue depth value is less than the issued communication value, and listing the first communication on the device's pending list when the device's queue depth value is less than the issued communication value.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure generally relates to computer networking, storage networking, and communications. Some embodiments of this disclosure relate to systems and methods for throttling or controlling input / output access and throughput in a network. Background Technology

[0002] Over the past few decades, the market for network communication devices has grown by orders of magnitude, driven by the use of portable devices and the increased connectivity and data transfer between a wide variety of devices. Digital switching technologies have facilitated the large-scale deployment of affordable, easy-to-use communication networks, including storage networks (e.g., Storage Area Networks (SANs)). Wireless communication can operate according to various standards, such as IEEE 802.11x, IEEE 802.11ad, IEEE 802.11ac, IEEE 802.11n, IEEE 802.11ah, IEEE 802.11aj, IEEE 802.16 and 802.16a, Bluetooth, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), and cellular technologies.

[0003] A SAN connects computer data storage devices to servers in a commercial data center. SANs can use the Fibre Channel (FC) protocol, a high-speed data transport protocol that provides ordered, lossless delivery of raw block data. When a network node or link carries more data than it can handle, that link becomes congested in the network, resulting in a reduction in overall network throughput (e.g., for other infrastructure devices such as Fibre Channel Host Bus Adapters (HBAs) or storage adapters, Ethernet Converged Network Adapters (CNAs), security adapters, or other communication devices). Standards provide protocols to reduce or prevent congestion on the network. Summary of the Invention

[0004] On one hand, this disclosure provides an apparatus comprising: a pending list buffer configured to store a pending list; a queue configured to store a plurality of issued communications; and a circuit system configured to receive a first queue depth value of the apparatus, wherein the first queue depth value indicates an amount of the queue available for the issued communications of the apparatus, wherein the circuit system is configured to issue a first communication on the pending list of the apparatus if the first queue depth value is greater than a second queue depth value, the second queue depth value being a stored value.

[0005] On the other hand, this disclosure provides an apparatus comprising: a pending list buffer configured to store a pending list including a plurality of pending communications; a queue buffer configured to store a plurality of issued communications; and a circuit system configured to issue a first communication at least in part in response to a queue depth value greater than an issued communication value, wherein the queue depth value indicates an amount of the queue buffer available for the issued communications from a first device, and the issued communication value indicates an amount of issued communications from the first device stored in the queue buffer.

[0006] On the other hand, this disclosure provides a controller comprising: a queue buffer configured to store a plurality of issued communications; and a circuit system configured to issue a first communication for translation when a queue depth value is less than an issued communication value, wherein the queue depth value indicates an amount of the queue buffer available for the issued communications from the device, and the issued communication value indicates an amount of issued communications from the device stored in the queue buffer. Attached Figure Description

[0007] The various objects, aspects, features, and advantages of this disclosure will become more apparent and better understood through a detailed description taken in conjunction with the accompanying drawings, in which like reference characters denote corresponding elements throughout. In the drawings, like reference numerals generally indicate identical, functionally similar, and / or structurally similar elements.

[0008] Figure 1 This is a schematic block diagram of a network including a memory controller configured to use input / output (I / O) device throttling techniques, according to some embodiments;

[0009] Figure 2 It is based on some embodiments of the invention. Figure 1 A schematic block diagram illustrating the register file used by the memory controller;

[0010] Figure 3 This demonstrates the relationship with, according to some embodiments. Figure 1 A schematic flowchart illustrating the throttling operation of an instance of inbound I / O device in a network similar to the network described herein;

[0011] Figure 4 This demonstrates the relationship with, according to some embodiments. Figure 1 The flowchart illustrates the instantiation operation in a network similar to the network described in the diagram.

[0012] Figure 5 This demonstrates the relationship with, according to some embodiments. Figure 1 The flowchart illustrates the instantaneous cancellation of a suspended operation in a network similar to the one described in the diagram. Detailed Implementation

[0013] (Several) The full text of the following IEEE standards (including any draft versions of such standards) is hereby incorporated by reference for all purposes and constitutes part of this disclosure: IEEE 802.3, IEEE 802.11x, IEEE 802.11ad, IEEE 802.11ah, IEEE 802.11aj, IEEE 802.16 and 802.16a, and IEEE 802.11ac. Furthermore, although this disclosure may reference certain aspects of these standards, it is in no way limited to these standards. Some standards may relate to Storage Area Networks (SANs) for connecting computer data storage devices to servers in commercial data centers. SANs may use Fibre Channel (FC) standards / protocols, Small Computer System Interface (SCSI) interface standards / protocols, Asynchronous Transfer Mode (ATM) protocols, and Synchronous Optical Networking Protocol (SONET), the full text of all of which is incorporated by reference herein.

[0014] Some embodiments of the systems and methods utilize throttling or access control of device input / output (I / O) communications in a network. Throttling can refer to a technique used to handle memory processing more efficiently, wherein the rate of memory requests is selectively adjusted. In some embodiments, throttling can control the amount of resources used when the system is processing I / O communication. In some embodiments, throttling causes the system to either process the I / O communication or delay the processing of the I / O communication to free up resources for more critical I / O communication. In some embodiments, the system may perform throttling to delay the processing of I / O operations for offloading I / O operations. In some embodiments, I / O throttling operations are dynamic and performed in hardware (e.g., without operating system (OS) device software intervention). Some embodiments achieve the advantages of device I / O access control without exposing the global host queue depth to the OS device driver. Some embodiments of the system and method dynamically change the host queue depth and provide per-device (Logical Device (LD) or Simple Disk Bundle (JBOD) granularity. The terms hardware and hardware circuitry can refer to any circuit system or circuitry that operates without OS device software intervention, including but not limited to processors, application-specific circuits, programmable logic devices, ASICs, and combinations thereof. In some embodiments, the hardware may be configured by or execute non-OS software. Circuit system can refer to any electronic circuitry.

[0015] In some embodiments, the storage controller includes a command dispatcher unit that performs per-device dynamic I / O throttling. In some embodiments, per-device dynamic I / O throttling can be controlled from an application. Per-device I / O throttling may refer to throttling that can be performed on a specific device rather than on all devices in the system at once. Per-device dynamic I / O throttling may refer to throttling that can be performed on a specific device rather than on all devices in the system at once, wherein the throttling parameters are variable and not fixed. In some embodiments, the device I / O throttling mechanism is used to provide device-level queue depth control to perform the following actions:

[0016] 1. Prevent transient runtime resources (such as Host Distributed Cluster List (HSGL) and hash slot elements, Internal Distributed Cluster List (ISGL) and / or Independent Redundant Array of Independent Disks (RAID) Distributed Cluster List (RSGL)) from being exhausted after the usage reaches a threshold;

[0017] 2. To prevent the system from becoming unusable when resources, such as ISGL resources, are fixed with dirty data and cannot be released;

[0018] 3. By limiting the queue depth of each device handle, resources can be evenly distributed across multiple device handles;

[0019] 4. Allows firmware to maintain multiple sets of queue depth settings based on the requester;

[0020] 5. Apply queue depth according to IO type (read or write); and / or

[0021] 6. Allow users to set limits on the number of active I / Os that a particular device handle should have (e.g., a device with a bootable OS may not require a very high queue depth).

[0022] In some embodiments, the per-device queue depth is not controlled by the OS device driver and can be dynamically changed at the per-device (LD or JBOD) level to achieve granularity. In some embodiments, the HSGL is created by the command dispatch unit to represent direct memory access (DMA) operations to / from host memory. In some embodiments, the RSGL is created by the RAID manager in the on-chip RAID controller to represent DMA operations to / from dynamic random access memory (DRAM). In some embodiments, the ISGL is created by the buffer manager in the on-chip RAID controller to represent DMA operations to / from DRAM.

[0023] Some embodiments relate to a method of communication in a network. The communication is controlled by a controller. The method includes: providing a first communication associated with a device for transmission; transmitting the first communication if the transmitted communication value is less than a queue depth value of the device; and listing the first communication on a pending list of the device if the queue depth value of the device is less than the transmitted communication value. The queue depth value indicates the amount of queue available for the transmitted communication of the device, and the transmitted communication value indicates the amount of communication of the device stored in the queue. Listing can refer to placement on a list, which can be represented in any form, including but not limited to head and tail pointer schemes, tables, etc. The placement can occur in any order (e.g., priority, receive time, transmit time, placement time, random, etc.). A pending list can refer to a list of communications (e.g., I / O communications) waiting to be transmitted to a queue (e.g., queue 230 configured to store communications for transmission). The pending list can be stored in any type of storage device or memory and can store any type of communication representation (e.g., I / O communications stored using a head and tail pointer scheme). In some embodiments, a pending list buffer may refer to a buffer used to store the pending list. A buffer may refer to any type of memory or storage device. Communication may refer to any data transfer, including but not limited to I / O communication. I / O communication may take any form, including but not limited to programmed I / O communication, interrupt-initiated communication, direct memory access (DMA) communication, host-to-device communication, and device-to-device communication. I / O communication may be read communication in which data is requested and write communication in which data is requested to be stored.

[0024] In some embodiments, the method further includes issuing the first communication if the queue depth value of the device is not greater than zero. In some embodiments, the method further includes listing the communication on the device's pending list if the device's pending count value is not zero. The pending count value indicates the number of communications on the pending list. In some embodiments, the pending list uses head and tail pointers to indicate communications on the pending list. In some embodiments, the method performs device-level throttling without intervention from the host operating system. In some embodiments, the method performs hardware-based device-level throttling. The queue depth of the device may refer to the amount of queue available for the issued communication of the device. In some embodiments, the queue depth value of the device may be stored in any type of memory or storage device (e.g., register, buffer, field, etc.) and may reflect the amount of queue available for the issued communication of the device. A queue may refer to any type of memory or storage device. In some embodiments, the queue stores identifiers of communications in a retrievable order (e.g., first-in, first-out, etc.). The transmitted communication value of a device can refer to a value that reflects the amount of communication of the device stored in the queue or the amount of usage of the queue by transmitted communication, and can be stored in any type of memory or storage device (e.g., registers, buffers, fields, etc.). The pending count value of a device can refer to a value that reflects the amount of communication of the device provided on the pending list, and can be stored in any type of memory or storage device (e.g., registers, buffers, fields, etc.).

[0025] In some embodiments, the method further includes: decrementing the sent communication value upon completion of the first communication; and sending a second communication from the pending list if the sent communication value is less than the queue depth value. In some embodiments, the method further includes sending a second communication from the pending list if a new queue depth value greater than the queue depth value is provided. In some embodiments, the method performs device-level throttling and disables device-level throttling if the queue depth value is set to zero. In some embodiments, the method is executed on a command dispatch unit.

[0026] Some embodiments relate to a memory controller. The memory controller includes: a suspend list buffer for storing a plurality of suspended communication identifiers, the suspend list buffer being coupled to the memory controller; a queue buffer for storing a plurality of issued communication identifiers, the queue buffer being coupled to the memory controller; and circuitry configured to perform device-level throttling. The circuitry is configured to issue a first communication for translation when a queue depth value is less than an issued communication value. The queue depth value indicates the amount of the queue buffer available for the issued communication identifiers from a first device, and the issued communication value indicates the amount of issued communication from the first device to the queue buffer.

[0027] In some embodiments, the circuitry is configured to list the first communication in the pending list buffer if the queue depth value of the device is less than the issued communication value. In some embodiments, the first communication is not translated when it is listed on the pending list. In some embodiments, the set of registers includes a tail pointer register and stores information associated with the first communication when it is listed on the pending list. In some embodiments, the circuitry is hardware circuitry. A register refers to any type of storage device or memory, including but not limited to temporary storage within processing hardware.

[0028] Some embodiments relate to a memory controller that further includes a set of registers, the set of registers including a first register for the queue depth of the first device and a second register for the transmitted communication value of the first device. In some embodiments, the set of registers includes a third register for the queue depth of the second device and a fourth register for the transmitted communication value of the second device.

[0029] Some embodiments relate to a method of communication in a network. The communication is controlled by a controller. The method includes: receiving a new queue depth value from the device by the controller; determining whether the new queue depth value is greater than a current queue depth value; and if the new queue depth value is greater than the current queue depth value, then issuing a first communication from the device's pending list. The queue depth value indicates the amount of queue available for the device to send the communication.

[0030] In some embodiments, the queue depth value is used for reading communication. In some embodiments, the queue depth value is used for writing communication.

[0031] Before discussing specific embodiments of this solution, it may be helpful to describe aspects of the operating environment and associated system components (e.g., hardware elements) in conjunction with the methods and systems described herein. The systems and methods described can be employed in various types of networks. Some embodiments of the network include or communicate with a SAN, security adapter, or Ethernet CNA. In simple terms, a network environment includes a wireless communication system comprising one or more access points, one or more wireless or wired communication devices, and network hardware components. Wireless or wired communication devices may include, for example, laptop computers, tablet computers, personal computers, wearable devices, vehicles (e.g., cars, drones, intelligent vehicles, robotic units, etc.), switches, routers, and / or cellular phone devices. The network environment may be an ad hoc network environment, an infrastructure wireless network environment, a wired network coupled to a wireless network, a subnet environment, etc. In some embodiments, the network environment includes a host bus adapter (HBA), a switching network, a target HBA, and a target host. In some embodiments, a network arranged as an FC SAN may be included. The aspects of the operating environment and components described above will become more apparent in the context of the systems and methods disclosed herein.

[0032] refer to Figure 1 Network 200 includes host 202, storage controller unit 204, and devices 206A through C. In some embodiments, storage controller unit 204 is any type of device and includes storage controller 220 and messaging interface 240. In some embodiments, storage controller 220 includes processing hardware 210, a pending list 236, a register file 300, and a queue 230. Host 202 includes an operating system. A memory controller (e.g., storage controller 200) may refer to any device that controls or processes communications (e.g., I / O communications) for communication across network 200. In some embodiments, the memory controller uses message parsing interface 240 to communicate.

[0033] Processing hardware 210 is implemented as a hardware engine, an ASIC built according to packet processing rules, a field-programmable gate array (FPGA) or logic device, a communication processor, or any other type and form of dedicated semiconductor logic or processing circuitry system capable of processing or supporting the communications described herein. In some embodiments, processing hardware 210 is a hardware device (e.g., an ASIC and dedicated circuitry system) for performing throttling operations that are independent of OS software. In some embodiments, processing hardware 210 may receive data from a software-based application executed by host 202 and devices 206A to C. In some embodiments, processing hardware 210 is a controller that performs operations in response to firmware or other software.

[0034] Network 200 can be used with various types of computing and communication components. For example, network 200 may include two or more devices communicating with each other via cable, wired, or wireless media. In some embodiments, network 200 includes a target host or host 202. In some embodiments, network 200 is a high-speed network connecting host 202 to a target or data storage device (e.g., a high-performance storage subsystem), such as devices 206A to C. In some embodiments, network 200 can access a storage processor (SP) and a storage disk array embodied in devices 206A to C. In some embodiments, devices 206A to C may be any type of electronic device participating in communication (e.g., I / O communication) and may be an LD or JBOD. Network 200 may include a switching network coupled between storage controller unit 204 and devices 206A to C.

[0035] A switching network includes receive ports, transmit ports, and buffers or crossover switches. In some embodiments, the switching network is a configured switching network, an arbitration ring network, or a point-to-point network. Receive ports and transmit ports may be associated with servers, hubs, switches, routers, directors, nodes, or other devices. The switching network may include a physical layer, interconnection devices, and translation devices.

[0036] In some embodiments, the storage controller unit 204 is a server storage integrated circuit-based product providing communication connectivity. In some embodiments, the storage controller unit 204 provides data center and storage operations for the Peripheral Component Interconnect Fast (PCIe) Gen4 standard expansion bus. In some embodiments, the storage controller unit 204 is a Serial Advanced Technology Attachment (SATA), Serial Attached SCSI (SAS) controller, SAS Redundant Array of Independent Disks (RAID) controller, and / or host bus adapter, or a portion thereof. In some embodiments, the storage controller unit 204 is a SAS I / O controller, an on-chip RAID (ROC) IC, and a SAS expander IC, or a portion thereof. In some embodiments, the storage controller unit 204 provides connectivity for directly attached storage devices, such as devices 206A to C (e.g., low-latency flash memory arrays, solid-state drives, Fibre Channel storage systems, and hard disk drive (HDD) arrays).

[0037] In some embodiments, the storage controller unit 204 is integrated with the host 202, a line card, a mezzanine card, a motherboard device, an IC package, a standalone device, or other devices. The storage controller unit 204 is provided in an integrated circuit package that includes processing hardware 210, a queue 230, a pending list 236, and a messaging interface 240. The messaging interface 240 communicates directly or indirectly (e.g., via a switching network or other means) with devices 206A to C. In some embodiments, the storage controller unit 204 translates Message Passing Interface (MPI) message frames on the messaging interface 240 into translated I / O and allocates resources for processing the message frames. In some embodiments, the I / O is translated into frames that can be processed by other hardware units in the storage controller unit 204. The storage controller unit 204 can be any type of device for processing data communication between the host 202 and devices 206A to C.

[0038] Message passing interface 240 is the interface between storage controller unit 204 and devices 206A to C. In some embodiments, message passing interface 240 operates according to a standardized and portable messaging MPI standard designed to function on parallel computing architectures. In some embodiments, the MPI standard defines the syntax and semantics of library routines useful to a wide range of users who write portable messaging programs in C, C++, and Fortran. In some embodiments, if device throttling is not required, storage controller unit 204 copies data from host 202 and allocates resources for translation. Data associated with I / O is provided across interface 240.

[0039] Queue 230 and pending list 236 are storage devices (e.g., memory) configured to store pending I / O and representations of pending I / O. I / O refers to I / O communication. In some embodiments, each issued I / O communication is stored as a device handle in an entry of queue 230, the device handle being a name used by the operating system to identify the storage resource and the correct means (e.g., a driver or system call) for accessing the storage resource. The entry may store additional information about the I / O connection. Queue 230 may be a separate queue or a combined queue for read and write operations. In some embodiments, queue 230 is provided for each of devices 206A through C. Queue 230 may be a separate read buffer and write buffer (FIFO buffer) for tracking issued I / O. In some embodiments, queue 230 is a general queue for all devices, where a separate queue depth is tracked for each device. Queue 230 and the buffer for pending list 236 may be a single storage device.

[0040] Suspended list 236 stores pending I / O communications awaiting issuance. A head and tail pointer mechanism is used to store I / Os in suspended list 236. In some embodiments, suspended list 236 is a per-device linked list, wherein the first and last I / Os of the linked list are indicated by a tail pointer and a head pointer, respectively. Each I / O includes a next pointer pointing to the next I / O in the linked list, and the next pointer to the last I / O (indicated by the tail pointer) points to a null reference, thus indicating the end of the list. In some embodiments, suspended list 236 is a per-device (devices 206A to C) linked list. In some embodiments, a separate suspended list exists for read and write operations on each device. In some embodiments, I / Os are not translated when issued from suspended list 236, and the command dispatch unit (CDU) of storage controller unit 204 translates the untranslated I / Os into frames understood by other hardware units in storage controller unit 204. Untranslated communication (e.g., I / O) may refer to communication (e.g., I / O) that has not yet been translated into frames suitable for processing by the hardware of storage controller unit 204. In some embodiments, translation may involve frame processing and protocol processing by a command dispatch unit. A head pointer may indicate the start of a linked list or the first communication, and a tail pointer may indicate the end of a linked list or the last communication. A head pointer register may refer to a register used to store an indication of the start of a linked list or the first communication, and a tail pointer register may refer to a register used to store an indication of the end of a linked list or the last communication. Pending communication may refer to communication stored on a pending list (e.g., pending list 236). In some embodiments, communication is stored in the pending list while awaiting an opportunity to be offered to a queue (e.g., queue 230) for transmission.

[0041] Register file 300 includes a set of registers configured to store information for I / O processing controlled by storage controller unit 204. Register file 300 may include a set of registers for each device 206A to C (e.g., a per-device register file). In some embodiments, a register file exists for read and write operations for each device. The following section discusses... Figure 2 Some embodiments of register file 300 are described in more detail.

[0042] Processing hardware 210 is configured to include a command dispatch unit and a throttling module 212 configured to implement per-device dynamic I / O throttling. In some embodiments, processing hardware 210 performs flow control operations faster than software-based OS operations. In some embodiments, processing hardware 210 is configured to allocate HSGLs among devices 206A to C to ensure that communication processing resources are not exhausted by one or both devices 206A to C. In some embodiments, module 212 is a dedicated hardware module or a combination of hardware and software that performs per-device throttling using register file 300 and pending list 236. Module 212 and processing hardware 210 are any circuitry or components capable of performing logic and communication processing for the throttling operations described herein. In some embodiments, module 212 and processing hardware 210 are implemented as a field-programmable gate array, application-specific integrated circuit, hardware, software execution processor, or state machine.

[0043] Processing hardware 210 uses queue 230 to buffer or store information for issued I / O connections. If the total number of incomplete I / Os (issued I / O count value) of the devices (devices 206A to C) has reached the queue depth limit (queue depth value) of the device, then processing hardware 210 suspends the untranslated I / Os. According to some embodiments, the limit for each device and the number of incomplete I / Os are tracked via register file 300, as discussed below. Processing hardware 210 suspends untranslated I / Os by placing them on the suspension list 236 in the order they are received (first in, first out). When one of the devices 206A to C completes an issued I / O via communication across message passing interface 240, the storage controller unit 204 cancels the suspension and issues the previously suspended I / O (if any). If the new queue depth value is higher than the previous (current) value, then the update of the queue depth value triggers the issuance of untranslated I / Os from the suspension list 236. Disabling queue depth (e.g., setting the maximum queue depth value or the queue depth to zero) causes the storage controller unit 204 to issue all IOs from the pending list 236.

[0044] refer to Figure 1 and 2 The processing hardware 210 and throttling module 212 use a set of registers in register file 300 for each of devices 206A to C to control the addition and removal of I / O from the pending list 236, and issue I / O according to device throttling provided by throttling module 212. Register file 300 can be provided for read queues and write queues. Figure 2The register file 300 provides an instance size, values, field names, and descriptions. In some embodiments, the register file 300 is a temporary storage device within the processing hardware 210. Issuing or issuance of communication (e.g., I / O) may refer to placing communication (e.g., I / O) or its identifier from a pending list or from another source (e.g., a source including the storage controller unit 204) into a queue for processing and / or ultimately delivered to the host 202 or via the message parsing interface 240. In some embodiments, I / O communication may be processed for delivery directly from the pending list 236 or from another source. Issued communication may refer to communication that has been experienced or has begun to be issued. Issued communication may refer to more than one communication that has been experienced or has begun to be issued. In some embodiments, issued communication or its identifier is stored in a queue until it is transmitted.

[0045] In some embodiments, register file 300 includes: a queue depth register 302 for storing queue depth values ​​of devices in devices 206A to C; a maximum queue depth register 304 for storing maximum queue depth values ​​of devices in devices 206A to C; a head pointer register 306 for storing a head pointer (head Ptr) pointing to the top of the pending list 236; a tail pointer register 308 for storing a tail pointer (tail Ptr) pointing to the bottom of the pending list 236; and a pending count register 310 for storing... The number of suspended I / Os in the suspended list 236 (suspend count value); the issued I / O count register 312, which stores the number of incomplete I / Os (issue count value); the I / O type register 314, which stores the I / O type value (read or write); the queue depth error register 316, which stores the queue depth error value (e.g., indicating an error when the queue depth is greater than the maximum queue depth); and the queue depth error mask register 318, which stores the queue depth error mask value (e.g., the value generated by an interrupt based on the queue depth error value).

[0046] In some embodiments, firmware can be used to control throttling of devices 206A to C by providing a desired value to register file 300 (e.g., a queue depth value in queue depth register 302). In some embodiments, a user-space application can set a maximum queue depth value in register 304 via firmware. In some embodiments, if not explicitly set by the application, the field is disabled (set to zero), thereby disabling device throttling. The queue depth value cannot exceed the maximum queue depth (if it is non-zero) at any time. Therefore, if throttling has been applied to the device and the queue depth value is greater than the maximum queue depth value set by the application, the queue depth value is set to the maximum queue depth value. In some embodiments, firmware activates, updates, and disables the device's queue depth value depending on system conditions and / or design criteria. Device applications can configure the maximum queue depth value or disable throttling by setting the maximum queue depth value to zero.

[0047] refer to Figures 1 to 3 The storage controller unit 204 performs inbound I / O throttling according to process 500. Process 400 is used to issue an incoming IOP or, for each of devices 206A to C, use the head Ptr, tail Ptr, pending count, and issued I / O count value in register file 300 to place the incoming IOP in pending list 236.

[0048] Process 400 includes operation 402 in which an incoming I / O (e.g., MPI I / O) is received. At operation 406, throttling module 212 determines whether the queue depth value in register 302 of the I / O type is greater than 0. If not, then the normal process proceeds at operation 408. The normal process involves directly issuing the I / O. If yes, then process 400 proceeds to operation 410.

[0049] At operation 410, the pending count value is received. If the pending count value in register 310 is zero, then process 400 proceeds to operation 412. If the pending count value in register 310 is not zero, then process 400 proceeds to operation 416. At operation 416, the pending count value in register 310 is incremented by 1 and the IO is added to the pending list 236. The IO is added to the pending list 236 by adjusting the tail pointer in register 308, the head pointer in register 306, and the next pointer in the adjacent IO to place the IO on the linked list defined in registers 306 and 308.

[0050] At operation 412, the issued I / O count value is received. If the issued I / O count value in register 312 is less than the queue depth value, then process 400 proceeds to operation 414. At operation 414, the issued I / O count value in register 312 is incremented by 1 and an I / O is issued. If the issued I / O count value in register 312 is greater than or equal to the queue depth value in register 302, then process 400 proceeds to operation 416. After operations 414 and 416, the storage controller unit 204 returns to operation 402.

[0051] refer to Figure 4 The storage controller unit 204 issues pending I / O from the pending list 236 according to process 500. In some embodiments, the storage controller unit 204 uses the pending list 236 and the head Ptr, tail Ptr, pending count, and issued I / O count values ​​in the register file 300 to execute process 500.

[0052] Process 500 includes operation 502, in which an indication of outgoing IO completion is received and the outgoing IO count in register 312 is decremented. At operation 504, throttling module 212 determines whether the outgoing IO count is less than the queue depth value and whether the pending count is greater than 0. If yes, then process 400 proceeds to operation 506. If not, then process 500 returns to operation 502 and waits for an indication of outgoing IO completion because another IO cannot be issued due to its absence or because the device's queue depth value has been exceeded.

[0053] At operation 506, the pending count in register 310 is decremented by 1 and the issued I / O count is incremented by 1. The I / O is issued and its identifier is removed from the list in pending list 236 by adjusting the tail pointer in register 308 (if necessary) and the head pointer in register 306.

[0054] refer to Figure 5 In the event of a change in queue depth, the storage controller unit 204 issues pending I / O from the pending list 236 according to process 600. In some embodiments, in process 600, the storage controller unit 204 uses the pending list 236 and the head Ptr, tail Ptr, pending count, and issued I / O count values ​​in the register file 300.

[0055] Process 600 includes operation 602, in which the queue depth value of the device is updated or disabled. At operation 604, throttling module 212 determines whether the new queue depth value for the I / O type is 0. If so, process 600 proceeds to operation 606. At operation 606, throttling is disabled. At operation 606, the queue depth value in register 302, the pending count value in register 310, and the issued I / O count value in register 312 are reset. At operation 606, all pending I / Os in the pending list 236 are issued.

[0056] If the new queue depth value for the IO type is not 0 at operation 604, then process 600 proceeds to operation 608. At operation 608, throttling module 212 determines whether the new queue depth value for the IO type is greater than the previous or current queue depth value in register 302. If the new queue depth value for the IO type is greater than the previous or current queue depth value in register 302, then process 600 proceeds to operation 612. If not, then process 600 proceeds to operation 610. At operation 612, the new queue depth value for the IO type is written to register 302 and IOs from the pending list 236 are issued until the pending list 236 is empty or the issued IO count equals the queue depth value. The pending count is decremented and the issued IO count is incremented based on the number of issued IOs. At operation 610, the new queue depth value for the IO type is written to register 302. After operations 610 and 612, the process returns to operation 602.

[0057] The construction and arrangement of the systems and methods shown in the various exemplary embodiments are merely illustrative. Although only a few embodiments are described in detail in this disclosure, many modifications are possible (e.g., variations in the number of ports or destinations, data types, methods of re-insertion, methods of reintroduction, parameter values, arrangements, etc.). For example, the positions of elements may be reversed or otherwise varied, the connections between elements may be direct or indirect, such that one or more intermediate elements may be present, and the nature or number of discrete elements or positions may be changed or varied. Therefore, all such modifications are intended to be included within the scope of this disclosure. According to alternative embodiments, the order or sequence of any process or method steps may be varied or reordered. Other substitutions, modifications, alterations, and omissions may be made in terms of the design, operating conditions, and arrangement of the exemplary embodiments without departing from the scope of this disclosure. For example, embodiments of this disclosure may be implemented by a single device and / or system or by a combination of single devices and / or systems.

[0058] This disclosure contemplates methods, systems, and program products on any machine-readable medium for implementing various operations. Embodiments of this disclosure may be implemented using existing computer processors, or by a dedicated computer processor incorporated for a suitable system for this or another purpose, or by a hardwired system. Embodiments within the scope of this disclosure include program products comprising machine-readable media for carrying or storing machine-executable instructions or data structures. This machine-readable medium may be any available medium accessible by a general-purpose or special-purpose computer or other machine having a processor. For example, this machine-readable medium may include RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can carry or store desired program code in the form of machine-executable instructions or data structures and is accessible by a general-purpose or special-purpose computer (e.g., ASIC or FPGA) or any other machine having a processor. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing machine to perform a function or group of functions.

[0059] Although the accompanying drawings illustrate a specific order of method steps, the order of steps may differ from the depicted order. Furthermore, two or more steps may be performed simultaneously or partially simultaneously. This variation will depend on the selected software and hardware system and the designer's choices. All such variations are within the scope of this disclosure. Similarly, software implementations may utilize standard programming techniques with rule-based logic and other logic for implementing various connection steps, processing steps, comparison steps, and decision steps.

[0060] When an element is referred to herein as “connected” or “coupled” to another element, it should be understood that the element may be directly connected to the other element, or that there may be an intermediary element between the elements. Conversely, when an element is referred to as “directly connected” or “directly coupled” to another element, it should be understood that there is no intermediary element in the “direct” connection between the elements. However, the presence of a direct connection does not preclude other connections in which intermediary elements may be present.

[0061] Furthermore, for ease of description, the methods and processes discussed herein may be described in a specific order. However, it should be understood that, unless the context otherwise requires, intermediate processes may occur before and / or after any part of the described process, and further processes may be reordered, added, and / or omitted according to various embodiments.

[0062] Unless otherwise indicated, all figures used herein to represent quantities, dimensions, etc., should be understood as subject to modification depending on system guidelines and application parameters. In this application, the use of the singular includes the plural unless specifically stated otherwise, and the terms “and” and “or” mean “and / or” unless otherwise indicated. Furthermore, the use of the term “including” and other forms (e.g., “includes” and “included”) should be considered non-exclusive. Moreover, terms such as “element” or “assembly” cover both elements and assemblies comprising one unit and elements and assemblies comprising more than one unit, unless specifically stated otherwise.

Claims

1. An apparatus comprising: A pending list buffer, configured to store a pending list for each device, wherein each pending list stores communications to be provided to a queue for transmission to the corresponding device; The queue is configured to store multiple outgoing communications to be transmitted to the corresponding device; and A first register is configured to store a queue depth value for a first device, wherein the queue depth value indicates the amount of the queue that can be used for communication initiated by the first device; The circuit system is configured as follows: If the first queue depth value of the first device is less than or equal to the transmitted communication value, then the first communication is listed in the pending list of the first device, wherein the first queue depth value is currently stored in the first register, and wherein the first communication is a command request of the first device. Receive the second queue depth value from the first device, and If the second queue depth value is greater than the first queue depth value, the first communication on the pending list of the first device is sent. as well as The second queue depth value is stored in the first register to replace the first queue depth value.

2. The device according to claim 1, wherein the queue depth value is used for reading communication or for writing communication.

3. An apparatus comprising: A pending list buffer, configured to store a pending list for each device, each pending list including multiple pending communications to be provided to a queue buffer for transmission to the corresponding device; The queue buffer is configured to store multiple outgoing communications to be transmitted to the corresponding device; A first register configured to store a queue depth value of a first device, wherein the queue depth value indicates the amount available for use in the queue from which communication has been initiated; and The circuit system is configured as follows: In response to the first device’s queue depth value being less than or equal to the sent communication value, the first communication is added to the pending list, wherein the first communication is a command request from the first device; The first communication is sent in at least part of the response that the queue depth value of the first device is greater than the sent communication value, wherein the sent communication value indicates the amount of sent communication of the first device stored in the queue buffer; If a new queue depth value greater than the previously stated queue depth value is provided, a second communication is initiated from the pending list; and The new queue depth value is stored in the first register to replace the original queue depth value.

4. The device of claim 3, wherein the circuitry is configured to perform throttling, wherein the first communication is not translated when it is listed on the pending list.

5. The device of claim 3, further comprising a set of registers, the set of registers including a tail pointer register configured to store information related to the first communication when listing the first communication on the pending list.

6. The device of claim 5, wherein the set of registers includes a third register configured to store the queue depth value of the second device and a fourth register configured to store the transmitted communication value of the second device.

7. The device of claim 3, wherein the circuit system is a hardware circuit configured as a memory controller.

8. The device according to claim 3, further comprising: A second register configured to store the transmitted communication value of the first device.

9. A controller comprising: A pending list buffer, configured to store a pending list; A queue buffer configured to store multiple outgoing communications to be transmitted to the device; A first register configured to store a queue depth value of the device, wherein the queue depth value indicates the amount that can be used for the queue buffer from which communication is initiated; and The circuit system is configured as follows: If the queue depth value of the device is less than or equal to the issued communication value, then the first communication is listed in the pending list, wherein the first communication is a command request of the device; If the queue depth value of the device is greater than the issued communication value, the first communication for translation is issued, wherein the issued communication value indicates the amount of issued communication of the device stored in the queue buffer; If a new queue depth value greater than the previously stated queue depth value is provided, a second communication is initiated from the pending list; and The new queue depth value is stored in the first register to replace the original queue depth value.

10. The controller of claim 9, wherein the circuitry is configured to send the first communication when the queue depth value of the device is not greater than zero.

11. The controller of claim 9, wherein the circuitry is configured to list the first communication on the device's pending list when the device's pending count value is not zero, the pending count value indicating the number of communications on the pending list.

12. The controller of claim 9, wherein the pending list uses a head pointer and a tail pointer to indicate communications on the pending list.

13. The controller of claim 9, wherein the circuitry is configured to perform hardware-based device-level throttling, the hardware-based device-level throttling being performed by: issuing the first communication if the issued communication value of the device is less than the queue depth value of the device; and listing the first communication on the device's pending list if the queue depth value of the device is less than or equal to the issued communication value.

14. The controller of claim 9, wherein the circuitry is configured to decrement the sent communication value when the first communication is completed, and to send a second communication from the pending list if the sent communication value is less than the queue depth value.

15. The controller according to claim 9, further comprising: A second register configured to store the transmitted communication value of the device.

16. The controller of claim 9, further comprising: A set of registers, including a tail pointer register configured to store information related to the first communication when the first communication is listed on the pending list.

17. The controller of claim 16, wherein the set of registers includes a third register configured to store the queue depth value of the other device and a fourth register configured to store the transmitted communication value of the other device.

Citation Information

Patent Citations

  • Queue depth management for communication between host and peripheral device

    CN101421791A