Dynamic power management among multiple storage devices

Abstract

This application relates to dynamic power management between multiple storage devices of an electronic system. The multiple storage devices are coupled into a storage device ring and pass power data packets along a power control path that continuously traces the storage device ring. During a current cycle, a first storage device receives the power data packet from an upstream storage device on the power control path, and the power data packet includes at least a system power level indicative of a total power consumption of the multiple storage devices. The first storage device sets a current power level of the first storage device based on the received power data packet, updates the power data packet based on the current power level, and sends the updated power data packet to a downstream storage device on the power control path.

Description

Technical Field

[0001] This application generally relates to memory management, including but not limited to methods, systems, and non-transitory computer-readable media for managing the power of storage devices in a storage system. Background Technology

[0002] Memory is used in computer systems to store instructions and data. Specifically, computer systems rely on non-volatile memory to retain the instructions and data stored thereon when the computer system is decoupled from power supply. Examples of auxiliary memory include, but are not limited to, hard disk drives (HDDs) and solid-state drives (SSDs). Different SSDs can be configured to perform different memory functions under the control of their host devices. Many electronic systems (e.g., servers, j-board bins, racks, application devices) include a large number (e.g., 4 to 30) of SSDs. Each of these SSDs varies in power utilization from about 5W at idle to about 25W at full power / performance today (and up to 40W in some cases) to meet the requirements of the high-speed peripheral component interconnect (PCIe) 6.0 standard. For example, an electronic system using 10 SSDs can have a power swing from 50W to 250W or up to 400W. This wide power swing of 50-400W requires compatible power supplies and thermal management designs in the electronic system. This problem only worsens when electronic systems include more SSDs and are used in data center racks and as part of the overall data center. Some solutions focus on over-designing the electronic system to accommodate worst-case scenarios where all SSDs operate at their maximum power levels. This incurs initial and recurring costs for these design decisions, even though this worst-case scenario rarely occurs during normal operation. Alternatively, some solutions set modest limits on the total power of the electronic system's SSDs at the expense of limiting the power of each individual SSD. The electronic system often operates at a compromise of power efficiency and does not allow for maximum performance on any single SSD. Developing mechanisms to manage the power of the storage devices in the electronic system to ensure efficient operation would be beneficial. Summary of the Invention

[0003] Various embodiments of this application relate to methods, systems, apparatuses, and non-transitory computer-readable media for dynamically managing the power of multiple storage devices coupled to a host device in an electronic system (e.g., a computer system). The electronic system sets a predefined power budget (also called a preferred power budget) for the total power of the multiple storage devices while still allowing each individual storage device to operate within its full power range. The total power consumption of the multiple storage devices is controlled based on the predefined power budget and distributed among the individual storage devices within their power ranges. Specifically, power data packets are transmitted between the individual storage devices using peer communication. The power data packets include at least a system power level indicating the total power consumption of the multiple storage devices. The multiple storage devices are arranged in an ordered storage device ring via the host device. When the power data packet is transmitted to each individual storage device on the ring, the corresponding storage device adjusts (e.g., increases, decreases) its own power level based on the system power level. For example, each storage device is configured to increase its power level based on a determination that the system power level is lower than the predefined power budget. In these ways, the system power level of the plurality of storage devices is maintained at or below the predefined power budget, and the power level of each individual storage device is dynamically adjusted within its power range based on whether the system power level reaches the predefined power budget.

[0004] In one aspect, a method is provided for dynamically managing the power of a plurality of storage devices coupled to a host device in an electronic system. The plurality of storage devices are coupled into a storage device ring. The method includes transmitting power data packets along a power control path that continuously tracks the storage device ring. Transmitting the power data packets along the power control path further includes, at a first storage device and during a current cycle: receiving the power data packet from an upstream storage device on the power control path; setting a current power level for the first storage device based on the received power data packet; updating the power data packet based on the current power level; and transmitting the updated power data packet to a downstream storage device on the power control path. The power data packet includes at least a system power level indicating the total power consumption of the plurality of storage devices.

[0005] In some embodiments, the power data packet further includes one or more of the following: a predefined power budget that defines an upper limit on the total power of the plurality of storage devices; a first power level of the first storage device during the most recent cycle prior to the current cycle; and a target power level of the first storage device during the current cycle.

[0006] In some embodiments, the method further includes: determining a first power level of the first storage device during the most recent cycle prior to the current cycle; identifying a target power level associated with the current cycle of the first storage device; and comparing the first power level of the first storage device with the target power level. Furthermore, in some embodiments, based on the determination that the first power level is greater than the target power level, the current power level of the first storage device is set to the target power level. Transmitting the power data packet further includes replacing the first power level of the first storage device with the current power level. Updating the power data packet further includes reducing the system power level of the power data packet by the amount of power change between the first power level of the first storage device and the target power level.

[0007] In another aspect, some embodiments of this application provide an electronic system including one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the processors to perform any of the methods described above on a storage system (e.g., including multiple SSDs).

[0008] In another aspect, some embodiments of this application provide a storage system including a plurality of storage devices (e.g., a plurality of SSDs) and a memory storing instructions thereon, the instructions causing the processors, when executed by the one or more processors, to perform any of the methods described above on the storage system.

[0009] In another aspect, some embodiments provide a non-transitory computer-readable storage medium storing one or more programs. The one or more programs include instructions that, when executed by one or more processors, cause the processors to perform any of the methods described above on a storage system (e.g., including multiple SSDs).

[0010] In some embodiments, the power data packets are communicated among the plurality of storage devices to fully utilize system characteristics where the workload is unevenly distributed across different storage devices when measured in seconds or minutes. The power levels of the storage devices change as they operate in idle, read, or write modes. For example, the power level of each storage device may swing between a first power (e.g., 5W) in idle mode and a second power (e.g., 25W) in write mode, with the second power level rarely being reached. It is almost impossible for all of the plurality of storage devices to operate in parallel at the second power. Thus, each individual storage device is allowed to use the second power when the system power level of the plurality of storage devices is controlled at or below a predefined power budget. The predefined power budget is less than the product of the second power and the total number of storage devices. This implements a cost-effective electronic system configured for normal operation and efficient power consumption.

[0011] In one example, a first electronic system has 10 SSDs, each with a power consumption cap of 25W. The total power consumption cap of the first electronic system is 250W. The electronic system is configured to provide and manage 250W of power to the 10 SSDs. A second electronic system has 10 SSDs, each with a power consumption cap of 12.5W. The total power consumption cap of the second electronic system is 125W. The performance of each SSD is limited by the aforementioned power consumption cap. Regardless of cost efficiency, the second electronic system impairs the performance of each of the 10 SSDs compared to the first electronic system. In contrast, in some embodiments of this application, a third electronic system has 10 SSDs, each with a power consumption cap of 25W. Given that each SSD consumes a very small amount (around 25W) and it is almost impossible for all SSDs to consume 25W simultaneously, the total power consumption of the third electronic system is capped (i.e., the predefined power budget) at 125W. The power consumption of each SSD is dynamically controlled to consume a maximum of 25W, while the total power consumption of the 10 SSDs is kept below 125W. This third electronic system achieves data storage performance similar to that of the first electronic system, while maintaining total power consumption at a reduced system power level and lowering power management requirements.

[0012] These illustrative embodiments and implementations are not intended to limit or restrict the invention, but rather to provide examples to aid in understanding the invention. Additional embodiments are discussed in the embodiments and further descriptions are provided therein. Attached Figure Description

[0013] To better understand the various described implementation schemes, the following diagrams should be used in conjunction with the implementation methods, where similar element symbols refer to corresponding parts throughout the diagrams.

[0014] Figure 1 This is a block diagram of an example system module in a typical electronic system according to some embodiments.

[0015] Figure 2 This is a block diagram of a storage system of an example electronic system having one or more memory access queues according to some embodiments.

[0016] Figure 3 This is a block diagram of an example electronic system according to some embodiments.

[0017] Figure 4A This is a block diagram of another example electronic system according to some embodiments, in which multiple storage devices transmit power data packets on a ring.

[0018] Figure 4B This is a block diagram of another example electronic system according to some embodiments, wherein a first storage device receives power data packets via a storage device ring.

[0019] Figure 5 A flowchart of an example program for dynamically managing the power of storage devices at each storage device in a storage device ring, according to some embodiments.

[0020] Figure 6 This is a flowchart of an example method for dynamically managing the power of a storage device in an electronic system, according to some embodiments.

[0021] In several views of the diagram, similar component symbols refer to the corresponding parts. Detailed Implementation

[0022] Reference will now be made in detail to specific embodiments, examples of which are illustrated in the accompanying drawings. Numerous non-limiting specific details are set forth in the following embodiments to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be practiced without these specific details. For example, it will be apparent to those skilled in the art that the subject matter presented herein can be implemented in many types of electronic systems or devices with data storage capabilities.

[0023] This application relates to dynamically managing the power of multiple storage devices coupled to a host device in an electronic system (e.g., a computer system). The electronic system sets a predefined power budget for the total power of the multiple storage devices while still allowing each individual storage device to operate within its full power range. The total power consumption of the multiple storage devices is controlled based on the predefined power budget and distributed among the individual storage devices within their power ranges. Specifically, power data packets are transmitted between the individual storage devices using peer communication. The power data packets include at least a system power level indicating the total power consumption of the multiple storage devices. The multiple storage devices are arranged in an ordered storage device ring via the host device. When the power data packet is transmitted to each individual storage device on the ring, the corresponding storage device adjusts (e.g., increases, decreases) its own power level based on the system power level. For example, each storage device is configured to increase its power level based on a determination that the system power level is lower than the predefined power budget. In these ways, the system power level of the plurality of storage devices is maintained at or below the predefined power budget, and the power level of each individual storage device is dynamically adjusted across its full power range based on whether the system power level reaches the predefined power budget.

[0024] Figure 1 This is a block diagram of an example system module 100 in a typical electronic system according to some embodiments. System module 100 in this electronic system includes at least: a processor module 102; a storage module 104 for storing programs, instructions, and data; an input / output (I / O) controller 106; one or more communication interfaces, such as a network interface 108; and one or more communication buses 140 for interconnecting these components. In some embodiments, the I / O controller 106 allows the processor module 102 to communicate with I / O devices (e.g., a keyboard, mouse, or trackpad) via a universal serial bus interface. In some embodiments, the network interface 108 includes one or more interfaces for Wi-Fi, Ethernet, and Bluetooth networks, each allowing the electronic system to exchange data with external sources, such as a server or another electronic system. In some embodiments, the communication bus 140 includes a circuitry (sometimes referred to as a chipset) that interconnects and controls communication between the various system components contained in system module 100.

[0025] In some embodiments, storage module 104 includes high-speed random access memory, such as DRAM, static random access memory (SRAM), double data rate (DDR) dynamic random access memory (RAM), or other random access solid-state storage devices. In some embodiments, storage module 104 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory storage devices, or other non-volatile solid-state storage devices. In some embodiments, storage module 104, or alternatively, the non-volatile storage device within storage module 104, includes a non-transitory computer-readable storage medium. In some embodiments, a storage slot is reserved on system module 100 for housing storage module 104. Once inserted into the storage slot, storage module 104 is integrated into system module 100.

[0026] In some embodiments, system module 100 further includes a component selected from one or more of a storage controller 110, an SSD 112, a hard disk drive (HDD) 114, a power management integrated circuit (PMIC) 118, a graphics module 120, and a sound module 122. The storage controller 110 is configured to control communication between the processor module 102 in the electronic system and the storage components including the storage module 104. The SSD 112 is configured to use an integrated circuit assembly to store data in the electronic system and, in many embodiments, is based on a NAND or NOR memory configuration. The HDD 114 is a conventional data storage device for storing and retrieving digital information based on an electromechanical disk. A power connector 116 is electrically coupled to receive external power. The PMIC 118 is configured to modulate the received external power to other desired DC voltage levels, such as 5 V, 3.3 V, or 1.8 V, as needed by various components or circuits within the electronic system (e.g., processor module 102). In some embodiments, the graphics module 120 is configured to generate an output image to feed to one or more display devices according to a desired image / video format of the display devices. In some embodiments, the audio module 122 is configured to facilitate the input of audio signals to and from an electronic system under the control of a computer program.

[0027] In some embodiments, system module 100 further includes an SSD 112' directly coupled to I / O controller 106. Instead, SSD 112 is coupled to communication bus 140. In one example, communication bus 140 operates according to a high-speed peripheral component interconnect (PCIe or PCI-E) standard, which is a serial extended bus standard for interconnecting processor module 102 to one or more peripheral devices and various system components including components 110 to 122, and controlling said one or more peripheral devices and various system components.

[0028] Furthermore, those skilled in the art will recognize the use of other non-transitory computer-readable storage media, as new data storage technologies have been developed for storing information in non-transitory computer-readable storage media within storage modules 104, SSDs 112 and 112', and hard disk drive 114. These new non-transitory computer-readable storage media include, but are not limited to, media made of biomaterials, nanowires, carbon nanotubes, and individual molecules, even if the corresponding data storage technology is currently under development and not yet commercialized.

[0029] Figure 2 This is a block diagram of a storage system 200 of an example electronic device having one or more memory access queues according to some embodiments. The storage system 200 is coupled to a host device 220 (e.g., ...). Figure 1 The processor module 102 is configured to store instructions and data for extended periods, such as when the electronic device is in sleep, hibernation, or powered off. The host device 220 is configured to access and process the instructions and data stored in the storage system 200 to execute the operating system and user applications. The storage system 200 further includes a controller 202 and a plurality of storage channels 204 (e.g., channels 204A, 204B, and 204N). Each storage channel 204 includes a plurality of storage units. The controller 202 is configured to execute firmware-level software to bridge the plurality of storage channels 204 to the host device 220.

[0030] Each memory channel 204 includes one or more memory packages 206 (e.g., two memory dies). In one example, each memory package 206 (e.g., memory package 206A or 206B) corresponds to a memory die. Each memory package 206 includes a plurality of memory planes 208, and each memory plane 208 further includes a plurality of memory pages 210. Each memory page 210 includes a set of ordered memory cells, and each memory cell is identified by a corresponding physical address. In some embodiments, the memory system 200 includes a plurality of superblocks. Each superblock includes a plurality of memory blocks, each of which further includes a plurality of memory pages 210. For each superblock, the plurality of memory blocks are configured to be written to and read from the memory system in parallel via a memory input / output (I / O) interface. Optionally, each superblock groups memory cells distributed across the plurality of memory planes 208, the plurality of memory channels 204, and the plurality of memory dies 206. In one example, each superblock includes at least one set of memory pages, each page being distributed across different memory dies 206, having the same die, plane, block, and page names, and accessed via different channels of the different memory dies 206. In another example, each superblock includes at least one set of memory blocks, each memory block being distributed across different memory dies 206, including multiple pages, having the same die, plane, and block names, and accessed via different channels of the different memory dies 206. The storage system 200 stores information about the ordered superblock list in a cache of the storage system 200. In some embodiments, the cache is managed by a host driver of the host device 220 and is referred to as the Host Management Cache (HMC).

[0031] In some embodiments, the storage system 200 includes single-cell (SLC) NAND flash memory chips, with each cell storing a single data bit. In some embodiments, the storage system 200 includes multi-cell (SLC) NAND flash memory chips, with each cell of the MLC NAND flash memory chip storing two data bits. In one example, each cell of the three-cell (TLC) NAND flash memory chip stores three data bits. In another example, each cell of the four-cell (QLC) NAND flash memory chip stores four data bits. In yet another example, each cell of the five-cell (PLC) NAND flash memory chip stores five data bits. In some embodiments, each cell may store any suitable number of data bits. Compared to non-SLC NAND flash memory chips (e.g., MLC SSDs, TLC SSDs, QLC SSDs, PLCSSDs), SSDs with SLC NAND flash memory chips operate at higher speeds, higher reliability, and longer lifespans; however, they have lower device density and higher prices.

[0032] Each memory channel 204 is coupled to a corresponding channel controller 214 (e.g., controller 214A, 214B, or 214N), which is configured to control internal and external requests for accessing memory cells in the corresponding memory channel 204. In some embodiments, each memory package 206 (e.g., each memory die) corresponds to a corresponding queue 216 for memory access requests (e.g., queue 216A, 216B, or 216N). In some embodiments, each memory channel 204 corresponds to a corresponding memory access request queue 216. Additionally, in some embodiments, each memory channel 204 corresponds to distinct and different memory access request queues 216. In some embodiments, a subset (less than all) of the plurality of memory channels 204 corresponds to distinct memory access request queues 216. In some embodiments, all of the plurality of memory channels 204 of the storage system 200 correspond to a single memory access request queue 216. Each memory access request may optionally be received internally by the storage system 200 to manage the corresponding memory channel 204, or received externally by a host device 220 to write or read data stored in the corresponding channel 204. Specifically, each memory access request includes one of the following: a system write request received from storage system 200 to write to the corresponding storage channel 204; a system read request received from storage system 200 to read from the corresponding storage channel 204; a host write request originating from host device 220 to write to the corresponding storage channel 204; and a host read request received from host device 220 to read from the corresponding storage channel 204. It should be noted that system read requests (also referred to as background read requests or non-host read requests) and system write requests are dispatched by the storage controller to implement internal memory management functions, including but not limited to scrap collection, wear leveling, read interference mitigation, memory snapshot capture, memory mirroring, caching, and memory spare.

[0033] In some embodiments, in addition to channel controller 214, controller 202 further includes a local storage processor 218, a host interface controller 222, an SRAM buffer 224, and a DRAM controller 226. The local storage processor 218 accesses multiple storage channels 204 based on one or more memory access request queues 216. In some embodiments, the local storage processor 218 writes to and reads from the multiple storage channels 204 based on storage blocks. Data from one or more storage blocks is jointly written to or read from the multiple channels. Data in the same storage block is not written in parallel via more than one operation. Each storage block optionally corresponds to one or more storage pages. In one example, each storage block to be jointly written to or read from the multiple storage channels 204 has a size of 16 KB (e.g., one storage page). In another example, each storage block to be jointly written to or read from the multiple storage channels 204 has a size of 64 KB (e.g., four storage pages). In some embodiments, each page has 16 KB of user data and 2 KB of metadata. Additionally, the number of storage blocks to be jointly accessed and the size of each storage block can be configured for each of the system read, host read, system write, and host write operations.

[0034] In some embodiments, the local storage processor 218 stores data to be written to or read from each of the plurality of storage channels 204 in an SRAM buffer 224 of the controller 202. Alternatively, in some embodiments, the local storage processor 218 stores data to be written to or read from each of the plurality of storage channels 204 in a DRAM buffer 228A, which is included in the storage system 200, for example by means of a DRAM controller 226. Alternatively, in some embodiments, the local storage processor 218 stores data to be written to or read from each of the plurality of storage channels 204 in a DRAM buffer 228B, which is a DRAM buffer provided by the processor module 102 (…). Figure 1 The main memory used. The local storage processor 218 of controller 202 accesses the DRAM buffer 228B via host interface controller 222.

[0035] In some embodiments, data in multiple memory channels 204 is grouped into coded blocks, and each coded block is called a codeword. For example, each codeword includes n bits, where k bits correspond to user data, and (nk) corresponds to integrity data of the user data, where k and n are positive integers. In some embodiments, the memory system 200 includes an integrity engine 230 (e.g., an LDPC engine) and a register 232 including multiple registers or SRAM cells or flip-flops and coupled to the integrity engine 230. The integrity engine 230 is coupled to the memory channels 204 via a channel controller 214 and an SRAM buffer 224. Specifically, in some embodiments, the integrity engine 230 has a data path connection to the SRAM buffer 224, which is further connected to the channel controller 214 via a data path controlled by a local memory processor 218. The integrity engine 230 is configured to verify the data integrity of each coded block of the memory channel 204.

[0036] Figure 3 This is a block diagram of an example electronic system 300 according to some embodiments. The electronic system 300 includes a host device 220 (e.g., Figure 1 The system includes a processor module 102 and a storage system 200 coupled to a host device 220. The storage system 200 is configured to store instructions and data for extended periods, such as when the electronic system 300 is in sleep, hibernation, or shutdown. The host device 220 is configured to access the instructions and data stored in the storage system 200 and process the instructions and data to execute an operating system and user applications. The storage system 200 further includes a plurality of storage devices 302. Each storage device 302 includes a storage controller 202 and one or more storage channels 204, each having a plurality of storage cells. The controller 202 is configured to execute firmware-level software to bridge the plurality of storage devices 302 to the host device 220. In one example, the storage devices 302 include eight storage channels, and each storage channel further includes eight memory dies 206. Each memory die 206 includes two storage planes 208 or arrays. Each storage plane 208 further includes a plurality of storage pages 210. Each storage page 210 includes a set of ordered storage units, and each storage unit is identified by a corresponding physical address.

[0037] In some embodiments, each of the plurality of storage devices 302 operates within a power range having a power limit P0 (e.g., 5W), and the plurality of storage devices 302 have a predefined power budget PB, which is less than a predefined portion of the sum of the power limits of all the plurality of storage devices 302. For example, the power limit P0 of each storage device 302 is 25W and corresponds to a memory write mode, such as where data is written in parallel to all memory dies 206 of the respective storage device 302. The plurality of storage devices 302 includes 10 storage devices and therefore has a sum of power limits equal to 250W. The predefined power budget PB is set to one of 100W, 125W, 150W, and 200W, which is less than 250W. More specifically, the predefined power budget PB is set to be less than 85% or 90% of the sum of the power limits of all the storage devices 302 of the storage system 200.

[0038] In some embodiments, electronic system 300 includes a power management module 310 configured to provide power to storage devices 304 of storage system 200. Electronic system 300 (e.g., a data server) never needs to operate all its storage devices (e.g., SSDs) simultaneously at their power limit P0; each individual storage device 302 needs to operate at its power limit P0. A predefined power budget PB is set to be lower than the sum of the power limits of all the multiple storage devices 302 without compromising the power limit P0 of each individual storage device 302. Power management module 310 can be reduced in size and manufactured at a lower cost to provide a predefined power budget PB for the multiple storage devices 302 as a whole and for the power limit P0 of each individual storage device 302.

[0039] In some embodiments, the electronic system 200 includes a set of storage devices 304. The plurality of storage devices 302 are a subset of the set of storage devices 304. The host device 220 selects a subset of the set of storage devices 302 as the plurality of storage devices 304. In some contexts, the host device 220 includes a system-level I / O controller 106. Figure 1 The plurality of storage devices 302 are arranged in a storage device ring (e.g., Figure 4A(Ring 440 in the middle). Power packets are transmitted along the power control path of the tracking storage device ring. Based on the direction of the power control path, upstream and downstream storage devices are assigned to each of the plurality of storage devices 302. In some embodiments, the plurality of storage devices 302 includes all storage devices in the storage system 200. In some embodiments, the storage system includes one or more remaining storage devices 306 that are different from the plurality of storage devices 302 and not arranged in the power control path. In some embodiments, the plurality of storage devices 302 form a first storage device ring, and a subset or all of the remaining storage devices 306 form a second storage device ring 308. Each storage device ring has a different predefined power budget PB and is configured to operate with a total power consumption not exceeding the different predefined power budget PB.

[0040] As power packets are transmitted along the storage device ring, a predefined power budget PB is dynamically distributed across multiple storage devices 302. The power consumption of each storage device 302 includes input / output (I / O) power that varies based on the type of the respective storage device 302 (e.g., TLC, QLC, PLC). For example, the I / O power of a PLC-based NAND flash memory chip is greater than that of a QLC-based NAND flash memory chip, which in turn is greater than that of a TLC-based NAND flash memory chip. The power consumption for read and write operations increases sequentially for SLC, MLC, TLC, QLC, and PLC-based NAND flash memory chips. In other words, the power consumption for read and write operations increases with the endurance level of the storage device 302. I / O power also varies with the type of data transmission protocol of the I / O interface. For example, a storage device using PCIe 5.0 configured to perform random read operations at a power limit of 25W P0 must be boosted for a storage device using PCIe 6.0 to perform random read operations. As the workload of storage device 302 increases (e.g., the workload includes more random write operations), the power consumption of storage device 302 increases. Additionally, as the drive capacity utilization of storage device 302 increases, the power consumption required for write operations by storage device 302 increases.

[0041] Figure 4A This is a block diagram of another example electronic system 300 according to some embodiments, wherein a plurality of storage devices 302 transmit power data packets 402 on a ring 440, and Figure 4BThe diagram illustrates another example of an electronic system 300 according to some embodiments, wherein a first storage device 302A receives a power data packet 402 via a storage device ring 440. The electronic system 300 is configured to dynamically manage the power consumption of a plurality of storage devices 302. The plurality of storage devices 302 are coupled into a storage device ring 440 (also referred to as a loop). The plurality of storage devices 302 have N storage devices (e.g., 302-1, 302-2, 302-3, ..., 302-N), where N is a positive integer greater than 1. In one example, storage device 302-1 is coupled to storage device 302-2, and storage device 302-2 is further coupled to storage device 302-3. Storage device 302-3 is coupled to storage device 302-4, and storage device 302-4 is further coupled to storage device 302-5. (See reference...) Figure 4A Storage device 302-3 is sequentially coupled to storage devices 302-4, 302-5, ... and 302-N. Storage device 302-N is coupled to storage device 302-1. Power data packet 402 is transmitted along power control path 404 of the continuously tracking storage device ring 440. Power data packet 402 includes at least a system power level PS indicating the total power consumption of the plurality of storage devices 302.

[0042] In some embodiments, the electronic system 300 includes a set of storage devices (e.g., Figure 2 (Storage device 304 in the group). A subset of the group of storage devices is selected as a plurality of storage devices 302 (e.g., 302-1, ..., 302-N) arranged in a storage device ring 440. An upstream storage device 302U and a downstream storage device 302D are assigned to each storage device 302 according to the power control path 404 of the tracking storage device ring 440. In some embodiments, each of the plurality of storage devices 302 receives a device information packet including information about the upstream storage device 302U and the downstream storage device 302D of the corresponding storage device 302 from the host device 220. Alternatively, in some embodiments, the initiating storage device 302S (e.g., 302-4) receives a device information packet 406 including information about the upstream storage device 302U and the downstream storage device 302D of each of the plurality of storage devices 302. The initiating storage device 302S transmits the device information packet 406 along the power control path 404 of the tracking storage device ring 440. Each of the remaining storage devices 302 identifies the corresponding upstream storage device 302U and the corresponding downstream storage device 302D based on the device information packet 406.

[0043] In some embodiments, each of the plurality of storage devices 302 operates within a power range having a power upper limit P0, and the plurality of storage devices 302 have a predefined power budget PB, which is less than a predefined portion of the sum of the power upper limits P0 of all the plurality of storage devices 302. In some cases, before receiving a power data packet 402, the initiating storage device 302S is initialized to set the system power level PS to the predefined power budget PB, and the plurality of device power levels of the plurality of storage devices 302 are set to be equal to each other. The sum of the plurality of device power levels is equal to the predefined power budget PB. The initiating storage device 302S updates the system power level PS and the device power levels in response to receiving the power data packet 402. Furthermore, in some embodiments, each of the plurality of storage devices 302 is initialized with a predefined power budget PB and equal device power levels, which are updated immediately upon receiving the power data packet 402. Alternatively, in some cases, before receiving the power data packet 402, the initiating storage device 302S is initialized to set the system power level PS to a predefined power budget PB, set the device power levels of a first subset of storage devices 302 to a power limit P0, and set the device power levels of the first subset of storage devices 302 to 0. The sum of the device power levels is equal to the predefined power budget PB. The initiating storage device 302S updates the system power level PS and its own current power level PC in response to receiving the power data packet 402. Furthermore, in some embodiments, each of a plurality of storage devices 302 is initialized with a predefined power budget PB and corresponding device power levels (e.g., power limit P0 or 0), which are updated immediately upon receiving the power data packet 402.

[0044] During the current cycle, the first storage device 302A receives a power data packet 402 from the upstream storage device 302U on the power control path 404. The power data packet 402 includes at least a system power level PS indicating the total power consumption of the plurality of storage devices 302. In some embodiments, the power data packet 402 includes a detailed power distribution of the system power level PS among the plurality of storage devices 302, i.e., multiple power levels of the plurality of storage devices. The first storage device 302A sets its current power level PC based on the received power data packet, updates the power data packet 402 based on the current power level PC, and sends the updated power data packet to the downstream storage device 302D on the power control path 404.

[0045] When the first storage device 302A receives a power data packet 402, its existing power level 408 (P1) is set during a previous cycle prior to the current cycle. In some scenarios, the first storage device 302A has a target operating mode corresponding to a target power level 410 during the current cycle, and the target power level 410 is greater than the existing power level 408 (P1) of the first storage device 302A. The first storage device 302A compares the system power level PS with a predefined power budget and determines whether the difference between the system power level PS and the predefined power budget covers the difference between the target power level 410 and the existing power level 408 (P1). Based on the determination that the difference between the system power level PS and the predefined power budget covers the difference between the target power level 410 and the existing power level 408 (P1), the current power level PC of the first storage device 302A is set to the target power level 410.

[0046] In some embodiments, the power data packet 402 includes the existing power level 408 (P1) of the first storage device 302A, and the first storage device 302A retrieves the existing power level 408 (P1) from the power data packet 402. Alternatively, in some embodiments, the power data packet 402 does not include the existing power level 408 (P1) of the first storage device 302A, and the first storage device 302A locally stores the existing power level 408 (P1) and retrieves the existing power level 408 (P1) from its local memory.

[0047] In some embodiments, the power data packet 402 further includes a predefined power budget PB that defines an upper limit on the total power of the plurality of storage devices 302. Alternatively, in some embodiments, the predefined power budget PB is provided separately from the power data packet 402 to each of the plurality of storage devices 302 and is stored locally in the respective storage device 302. In some embodiments, the power data packet 402 further includes a first power level of a first storage device 302A set during the most recent cycle prior to the current cycle. The existing power level 408 (P1) of the first storage device 302A when receiving the power data packet 402 is equal to the first power level of the first storage device 302A. Alternatively, in some embodiments, the first power level of the first storage device 302A is stored locally in the first storage device 302A. In some embodiments, the power data packet 402 further includes a target power level 410 of the first storage device 302A during the current cycle. Alternatively, in some embodiments, the target power level 410 of the first storage device 302A is stored locally in the first storage device 302A. Additionally, the power level of each of the remaining storage devices on ring 440 is set during the most recent cycle, optionally received via power data packet 402 or stored locally in the respective storage device, thus serving as the target power level 410 for each of the remaining storage devices on ring 440.

[0048] In some embodiments not shown in the figures, the storage device ring 440 further includes a host device 220. The host device 220 is coupled downstream of a second storage device and upstream of a third storage device on the power control path 404. The host device 220 monitors the system power level PS and the multiple device power levels of the plurality of storage devices 302. Alternatively, in some embodiments, one of the host devices 220 is coupled to each of the plurality of storage devices 302 and monitors the system power level PS and the multiple device power levels.

[0049] In some embodiments, a predefined power budget PB is increased to an updated power budget RBR based on a determination that a power shortage condition is met. The system power level PS is configured to change below the predefined power budget updated to the updated power budget PBR. Furthermore, in some embodiments, the updated power budget RBR is provided by a host device 220 coupled to one of the plurality of storage devices 302 (e.g., 302-2). Alternatively, in some embodiments, the predefined power budget is automatically increased by a predefined budget increase (e.g., 5W) or scaled by a predefined power scaler (e.g., 1.1) to produce the updated power budget PBR. Additionally, in some embodiments, the power shortage condition includes at least one of the following: the system power level PS has been maintained at the predefined power budget threshold duration; at least a predefined portion of the plurality of storage devices 302 has failed to increase the corresponding power level during a previous cycle of delivering power packets 402; and the first storage device 302A has failed to increase the first power level for at least a first number of cycles.

[0050] In some embodiments, a power shortage condition is identified locally (e.g., via storage controller 202) at any of a plurality of storage devices 302, and the storage device reports the power shortage condition to the host device 220. Alternatively, in some embodiments, a plurality of power state parameters 412A to 412C are monitored and cyclically cycled on power control path 404. For example, the plurality of power state parameters include, but are not limited to: a count 412A of storage devices 302 that failed to increase their power level during a previous cycle, a count 412B of storage devices 302 that failed to increase their power level, and a count 412C of the system power level PS remaining within a predefined power budget for a cycle. The storage device 302 located on power control path 404 identifies a power shortage condition based on the plurality of power state parameters 412A to 412C, updates the power budget level PB, and sends a command to update the power budget level PB to ring 440. Optionally, the storage device 302 located on the power control path 404 is a fixed storage device 302 (e.g., initiating storage device 302S) or any one of multiple storage devices 302. Alternatively, in some embodiments, the storage device 302 reports multiple power status parameters 412A to 412C to the host device 220, which identifies underpower conditions and updates the power budget level PB. The host device 220 may optionally send instructions to update the power budget level PB to the ring 440 via the fixed storage device 302 (e.g., initiating storage device 302S) or via more than one or all of the multiple storage devices 302.

[0051] In some embodiments, power data packets 402 are communicated among the plurality of storage devices to fully utilize system characteristics where the workload is unevenly distributed across different storage devices 302 when measured in seconds or minutes. The power level of storage devices 302 changes as they operate in idle, read, or write modes. For example, the power level of each storage device 302 may swing between a first power (e.g., 5W) in idle mode and a second power (e.g., 25W) in write mode, with the second power level rarely being reached. It is almost impossible for all of the plurality of storage devices 302 to operate in parallel at the second power. Thus, each individual storage device 302 is allowed to use the second power when the system power level PS of the plurality of storage devices 302 is controlled at or below a predefined power budget PB. The predefined power budget PB is less than the product of the second power and the total number of storage devices 302. This implements a cost-effective electronic system configured for normal operation and efficient power consumption.

[0052] In one example, a first electronic system has 10 SSDs, each with a power consumption capped at 25W. The total power consumption capped for the first electronic system is 250W. The electronic system is configured to provide and manage 250W of power to the 10 SSDs. A second electronic system has 10 SSDs, each with a power consumption capped at 12.5W. The total power consumption capped for the second electronic system is 125W. The performance of each SSD is limited by the aforementioned power consumption capped for each SSD. Regardless of cost efficiency, the second electronic system impairs the performance of each of the 10 SSDs compared to the first electronic system. In contrast, in some embodiments of this application, a third electronic system (e.g., Figure 4A The third electronic system (300) has 10 SSDs, each with a power consumption cap of 25W. Given that each SSD rarely consumes 25W and it is almost impossible for all SSDs to consume 25W simultaneously, an upper limit (i.e., a predefined power budget) for the total power consumption of the third electronic system is set at 125W, and the power consumption of each SSD is dynamically controlled to consume at most 25W, while the total power consumption of the 10 SSDs is kept below 125W. The third electronic system achieves data storage performance similar to that of the first electronic system, while maintaining total power consumption at a reduced system power level and reducing power management requirements.

[0053] In some embodiments, host device 220 configures SSD ring 440. SSD ring 440 may optionally include a subset or all of the SSDs of electronic system 300 (e.g., server). Host device 220 sets a power budget level PB (also known as maximum ring power MaxRingPower) to be applied on SSD ring 404. SSD ring 440 needs to be standardized for full deployment using NVMe or PCI-SIG. Error states default to a power limit for each SSD equal to the maximum ring power divided by the total number of SSDs in ring 440. Conservative error states are defined to keep the total power consumption of ring 440 within the power budget level PB, without all SSDs being able to exercise their power limit P0. During normal operation, power packets 402 (also known as power packets) are passed from SSD to SSD within ring 440 using PCIe peer communication. In some scenarios, power packet 402 is transmitted between two SSDs in 100 µsec, and across 10 SSDs in a loop in 1 msec. In one instance, power packet 402 arrives at the first SSD with an existing power level 408 (P1) and requests a target power level 410. Power packet 402 includes a system power level evaluated against a power budget level PB (maximum ring power). The first SSD adjusts its power level based on the difference between the system power level and the power budget level PB.

[0054] The performance of dynamic power control of storage system 200 is associated with one or more of the following: ring latency, SSD duty cycle, burst response time, workload characteristics, workload on each individual drive, maximum required server performance, I / O size for read and write operations, ratio between read and write operations, SSD I / O duty cycle during maximum server load, maximum number of ramp-up and ramp-down cycles from idle to I / O, synchronization of SSD workload across the server, acceptable burst power or time on maximum ring power, and SSD fairness. In some embodiments, the power budget level PB is divided into two packets with a 180-degree phase shift in ring 440.

[0055] refer to Figure 4BIn some embodiments, multiple storage devices 302 are coupled to a storage device ring 440. A power data packet 402 is transmitted along a power control path 404 that continuously tracks the storage device ring 440. During the current cycle of power data packet 402 transmission on the ring 440, a first storage device 302A receives power data packet 402 from an upstream storage device 302U on the power control path 404. The power data packet 402 includes at least a system power level PS indicating the total power consumption of the multiple storage devices 302. The first storage device 302A sets its current power level PC based on the received power data packet, updates the power data packet 402 based on the current power level PC, and sends the updated power data packet 402 to a downstream storage device 302D on the power control path 404.

[0056] In some embodiments, the first storage device 302A determines a first power level 408 during the most recent cycle prior to the current cycle. The first power level 408 may be provided by a power data packet 402 or retrieved locally from the memory of the first storage device 302A. The first storage device 302A identifies a target power level 410 associated with the current cycle of the first storage device 302A and compares the first power level 408 with the target power level 410. Furthermore, in some embodiments, based on the determination that the first power level 408 is greater than the target power level 410, the current power level PC of the first storage device is set to the target power level 410. The first power level 408 of the first storage device 302A is replaced with the current power level PC. The system power level PS of the power data packet 402 is reduced by the amount of power change between the first power level 408 and the target power level 410 of the first storage device 302A. The power data packet 402 is updated accordingly and passed to the downstream storage device 302D. Alternatively, in some embodiments, based on the determination that the first power level 408 is less than the target power level 410, the current power level PC of the first storage device 302A is set based on the system power level PS and the predefined power budget PB.

[0057] Specifically, in some embodiments, the first storage device 302A determines a first power level 408 set during the most recent cycle prior to the current cycle and identifies a target power level 410 associated with the current cycle of the first storage device 302A. The first storage device 302A determines whether a first difference between the first power level 408 and the target power level 410 is greater than a second difference between the system power level PS and a predefined power budget PB. Furthermore, in some scenarios, based on the determination that the first difference is less than the second difference (i.e., the remaining power budget is sufficient to achieve the target power level 410), the current power level PC of the first storage device 302A is set to the target power level 410. Additionally, in some embodiments, the first power level 408 of the power data packet is replaced with the current power level PC. The power data packet 402 is updated to include the system power level PS, which increases the power change between the first power level 408 and the target power level 410 of the first storage device 302A.

[0058] Conversely, in some scenarios, based on the determination that a first difference is greater than a second difference (i.e., the remaining power budget is insufficient to achieve the target power level 410), the current power level PC of the first storage device is set to the existing and applied first power level 408, and the updated power data packet further includes maintaining the system power level of the power data packet. The power data packet remains completely unchanged. Conversely, in some scenarios, based on the determination that a first difference is greater than a second difference (i.e., the remaining power budget is insufficient to achieve the target power level 410), the current power level PC of the first storage device 302A is set to the sum of the first power level and the second difference. Any remaining power budget is used entirely to increase the power level of the first storage device 302A, but the power level of the first storage device 302A still does not rise to the target power level 410. Therefore, in some embodiments, the first power level 408 of the first storage device 302A is replaced by the current power level PC, and the power data packet 402 is updated such that the system power level PS of the power data packet 402 increases to a predefined power budget PB.

[0059] Figure 5This is a flowchart of an example procedure 500 for dynamically managing the power of storage device 302 at each storage device 302 (e.g., first storage device 302A) in a ring 440 of storage device 302 according to some embodiments. During the current cycle in which power data packets 402 are transmitted on ring 440, first storage device 302A receives power data packets 402. First storage device 302A determines a first power level 408 (P1) (also referred to as the current power level 408) set during the most recent cycle prior to the current cycle and identifies a target power level 410 (PT) associated with the current cycle of first storage device 302A. First storage device 302A compares (operation 502) the first power level 408 (P1) with the target power level 410 (PT). Based on the determination that the first power level 408 (P1) is greater than the target power level 410 (PT) (operation 504), the current power level PC of first storage device 302A is set (operation 506) to the target power level 410 (PT). Replace the first power level 408 (P1) of the first storage device 302A with the current power level PC (operation 508). Reduce the system power level PS of the power data packet 402 (operation 510) by the power change (P1-PT) between the first power level 408 (P1) of the first storage device 302A and the target power level 410 (PT). The power data packet 402 is updated accordingly and passed to the downstream storage device 302D. Alternatively, in some embodiments, based on the determination that the first power level 408 (P1) is less than the target power level 410 (PT) (operation 512), the current power level PC of the first storage device 302A is set based on the system power level PS and a predefined power budget PB.

[0060] Specifically, in some embodiments, the first storage device 302A determines (operation 514) whether a first difference D1 between the first power level 408 (P1) of the first storage device 302A and the target power level 410 (PT) is greater than a second difference D2 between the system power level PS and the predefined power budget PB. Furthermore, in some scenarios, based on the determination that the first difference D1 is less than (operation 516) the second difference D2 (i.e., the remaining power budget is sufficient to achieve the target power level 410), the current power level PC of the first storage device 302A is set (operation 518) to the target power level 410 (PT). Additionally, in some embodiments, the first power level 408 (P1) of the power data packet is replaced (operation 520) with the current power level PC. The power data packet 402 is updated to include the system power level PS, which increases (operation 522) the power change (i.e., D1) between the first power level 408 (P1) of the first storage device 302A and the target power level 410 (PT).

[0061] Conversely, in some scenarios, based on the determination that the first difference D1 is greater than the second difference D2 (i.e., the remaining power budget is insufficient to achieve the target power level 410) (operation 524), the current power level PC of the first storage device 302A is set (operation 526) to the existing and applied first power level 408 (P1), and the updated power data packet further includes maintaining the system power level PS of the power data packet 402. The power data packet remains completely unchanged. Conversely, in some scenarios, based on the determination that the first difference is greater than the second difference (i.e., the remaining power budget is insufficient to achieve the target power level 410) (operation 524), the current power level PC of the first storage device 302A is set (operation 528) to the sum of the first power level 408 (P1) and the second difference D2. Any remaining power budget is used entirely to increase the power level of the first storage device 302A to the current power level PC, but the power level of the first storage device 302A still does not rise to the target power level 410 (PT). Therefore, in some embodiments, the first power level 408 (P1) of the first storage device 302A is replaced by the current power level PC (operation 530), and the power data packet 402 is updated such that the system power level PS of the power data packet 402 is increased (operation 532) to the predefined power budget PB.

[0062] Figure 6 This is a flowchart of an example method 600 for dynamically managing the power of a storage device 302 in an electronic system 300, according to some embodiments. The electronic system further includes a host device 220 and a storage system 200 coupled to the host device 220, and the storage system 200 includes a plurality of storage devices 302 (e.g., SSDs). The plurality of storage devices 302 are coupled into a storage device ring 440. The plurality of storage devices 302 transmit (operation 602) power data packets 402 along a power control path 404 that continuously tracks the storage device ring 440. During the current cycle 604, a first storage device 302A receives (operation 606) a power data packet 402 from an upstream storage device 302U on the power control path 404, and the power data packet 402 includes at least (operation 608) a system power level PS indicating the total power consumption of the plurality of storage devices 302. First storage device 302A sets (operation 610) its current power level PC based on the received power data packet 402, and updates (operation 612) the power data packet 402 based on the current power level PC. First storage device 302A then sends (operation 614) the updated power data packet 402 to downstream storage device 302D on power control path 404. (See above for reference.) Figure 3 as well as Figures 4A to 4B Further details about the storage device ring 400 are discussed.

[0063] In some embodiments, the power data packet 402 further includes (operation 616) one or more of the following: a predefined power budget PB defining an upper limit on the total power of the plurality of storage devices 302; a first power level P1 of the first storage device 302A during the most recent cycle prior to the current cycle; and a target power level PT of the first storage device 302A during the current cycle. Alternatively, in some embodiments, the predefined power budget PB is stored locally and individually on each of a subset or all of the plurality of storage devices 302. In some embodiments, for each of a subset or all of the plurality of storage devices 302 (e.g., the first storage device 302A), the corresponding power level or the corresponding target power level PT of the corresponding storage device is continuously and individually stored on the corresponding storage device. Specifically, in some embodiments, the first power level P1 or the target power level PT of the first storage device 302A is stored locally and individually on the first storage device 302A.

[0064] In some embodiments, the first storage device 302A determines a first power level P1 set during a previous cycle (e.g., the most recent cycle) preceding the current cycle, identifies a target power level PT associated with the current cycle of the first storage device 302A, and compares the first power level P1 of the first storage device 302A with the target power level PT. Furthermore, in some embodiments, based on the determination that the first power level P1 is greater than the target power level PT, the current power level PC of the first storage device 302A is set to the target power level PT. The first electronic system replaces the first power level P1 of the first storage device 302A with the current power level PC. Updating the power data packet 402 further includes reducing the system power level PS of the power data packet 402 by the amount of power change between the first power level P1 and the target power level PT of the first storage device 302A. Alternatively, in some embodiments, based on the determination that the first power level P1 is less than the target power level PT, the current power level PC of the first storage device 302A is set based on the system power level PS and a predefined power budget PB.

[0065] In some embodiments, setting the current power level PC of the first storage device 302A further includes: determining (operation 618) a first power level P1 of the first storage device 302A during the most recent cycle prior to the current cycle; identifying (operation 620) a target power level PT associated with the current cycle of the first storage device 302A; and determining (operation 622) whether a first difference between the first power level P1 of the first storage device 302A and the target power level PT is greater than a second difference between the system power level PS and a predefined power budget PB. Furthermore, in some embodiments, based on the determination that the first difference is less than the second difference, the current power level PC of the first storage device 302A is set to the target power level PT. Additionally, in some embodiments, the first storage device 302A replaces the first power level P1 of the power data packet 402 with the current power level PC. Updating the power data packet 402 further includes increasing the system power level PS of the power data packet 402 by the amount of power change between the first power level P1 of the first storage device 302A and the target power level PT.

[0066] In some embodiments, based on the determination that a first difference is greater than a second difference, the current power level PC of the first storage device 302A is set to a first power level P1, and updating the power data packet 402 further includes maintaining the system power level PS of the power data packet 402. Conversely, in some embodiments, based on the determination that a first difference is greater than a second difference, the current power level PC of the first storage device 302A is set to the sum of the first power level P1 and the second difference. Additionally, in some embodiments, the first storage device 302A replaces its first power level P1 with the current power level PC. Updating the power data packet 402 includes increasing the system power level PS of the power data packet 402 to a predefined power budget PB.

[0067] In some embodiments, the storage device ring 440 further includes a host device, a second storage device, and a third storage device, with the host device coupled downstream of the second storage device and upstream of the third storage device on a power control path 404. The host device monitors the system power level PS and the power levels of the plurality of storage devices 302.

[0068] In some embodiments, based on the determination that a power shortage condition is met, the electronic system increases a predefined power budget PB to an updated power budget. The system power level PS is configured to change below the predefined power budget PB increased to the updated power budget. Furthermore, in some embodiments, the storage system obtains the updated power budget from a host module coupled to one of the plurality of storage devices 302. In some embodiments, the storage system automatically increases the predefined power budget PB by a predefined budget increment to generate the updated power budget. Additionally, in some embodiments, the power shortage condition includes at least one of the following: the system power level PS has been maintained at the predefined power budget PB for a specified duration; at least a predefined portion of the plurality of storage devices 302 fails to increase the corresponding power level during a cycle of delivering power data packets 402; and the first storage device 302A fails to increase the first power level P1 within a first number of cycles.

[0069] In some embodiments, the electronic system includes a set of storage devices. The electronic system selects a subset of the set of storage devices as a plurality of storage devices 302, and arranges the plurality of storage devices 302 into a storage device ring 440 by assigning at least an upstream storage device and a downstream storage device to a first storage device 302A.

[0070] In some embodiments, each of the plurality of storage devices 302 operates within a power range having a power limit, and the plurality of storage devices 302 have a predefined power budget PB, which is less than a predefined portion of the sum of the power limits of all the plurality of storage devices 302. Furthermore, in some embodiments, the storage device 302S is initiated before the power data packet 402 is transmitted along the power control path 404. Figure 4A The system power level PS is set to a predefined power budget PB, and the power levels of multiple devices in the multiple storage devices 302 are set to be equal to each other. The sum of the power levels of the multiple devices is equal to the predefined power budget PB.

[0071] The memory also stores instructions and data associated with method 600, and includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state storage devices; and optionally includes non-volatile memory, such as one or more disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. The memory may optionally include one or more storage devices located remotely to one or more processing units. The memory, or alternatively, non-volatile memory within the memory, includes a non-transitory computer-readable storage medium. In some embodiments, the memory or the non-transitory computer-readable storage medium of the memory stores programs, modules, and data structures, subsets, or supersets for implementing method 600.

[0072] Each of the elements identified above may be stored in one or more of the previously mentioned storage devices and corresponds to a set of instructions for performing the functions described above. The modules or programs identified above (e.g., instruction sets) need not be implemented as separate software programs, procedures, modules, or data structures, and therefore, in various embodiments, various subsets of these modules may be combined or otherwise rearranged. In some embodiments, the memory may optionally store a subset of the modules and data structures identified above. Furthermore, the memory may optionally store additional modules and data structures not described above.

[0073] The terminology used in the descriptions of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the descriptions of the various embodiments described and the appended claims, the singular forms “a” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It should be further understood that the terms “comprising” and / or “including”, when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, it should be understood that although the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another.

[0074] As used herein, depending on the context, the term "if" may optionally be interpreted as meaning "when," "after," "in response to determination," "in response to detection," or "according to determination." Similarly, depending on the context, the phrase "if determination" or "if [the stated condition or event] is detected" may optionally be interpreted as meaning "after determination," "in response to determination," "after detection of [the stated condition or event]," "in response to detection of [the stated condition or event]," or "according to determination of detection of [the stated condition or event]."

[0075] For illustrative purposes, the foregoing description has been described with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the claims to the precise form disclosed. Many modifications and variations are possible in light of the foregoing teachings. The embodiments were chosen and described to best explain the operating principles and practical applications, thereby enabling those skilled in the art to understand.

[0076] Although various diagrams depict several logical stages in a specific order, stages independent of order can be reordered and other stages can be combined or decomposed. While some reorderings or other groupings are specifically mentioned, other reorderings or groupings will be obvious to those skilled in the art, and therefore the orderings and groupings presented herein are not an exhaustive list of alternatives. Furthermore, it should be recognized that the stages can be implemented in hardware, firmware, software, or any combination thereof.

Claims

1. A method for dynamic power management, comprising: At multiple storage devices coupled to the storage device ring: Power data packets are delivered along the power control path that continuously tracks the storage device ring, including at the first storage device and during the current cycle: The power data packet is received from an upstream storage device on the power control path, wherein the power data packet includes at least a system power level indicating the total power consumption of the plurality of storage devices; The current power level of the first storage device is set based on the received power data packet; Update the power data packet based on the current power level; as well as The updated power data packet is sent to the downstream storage device on the power control path.

2. The method of claim 1, wherein the power data packet further comprises one or more of the following: A predefined power budget defines the upper limit of the total power of the plurality of storage devices; The first power level of the first storage device during the most recent cycle prior to the current cycle; and The target power level of the first storage device during the current cycle.

3. The method according to claim 1, further comprising: Determine the first power level of the first storage device set during the previous cycle preceding the current cycle; Identify the target power level associated with the current cycle of the first storage device; as well as Compare the first power level of the first storage device with the target power level.

4. The method according to claim 3, wherein: Based on the determination that the first power level is greater than the target power level, the current power level of the first storage device is set to the target power level; Transmitting the power data packet further includes replacing the first power level of the first storage device with the current power level; and Updating the power data packet further includes reducing the system power level of the power data packet by the amount of power change between the first power level of the first storage device and the target power level.

5. The method of claim 3, wherein the current power level of the first storage device is set based on the system power level and a predefined power budget, according to the determination that the first power level is less than the target power level.

6. The method according to claim 5, further comprising setting the current power level of the first storage device as follows: Determine the first power level of the first storage device during the most recent cycle prior to the current cycle; Identify the target power level associated with the current cycle of the first storage device; as well as Determine whether a first difference between the first power level of the first storage device and the target power level is greater than a second difference between the system power level and the predefined power budget.

7. The method of claim 6, wherein the current power level of the first storage device is set to the target power level based on the determination that the first difference is less than the second difference.

8. The method of claim 7, further comprising: Replace the first power level of the power data packet with the current power level, wherein updating the power data packet further includes increasing the system power level of the power data packet by the amount of power change between the first power level of the first storage device and the target power level.

9. The method of claim 6, wherein setting the current power level of the first storage device to the first power level based on the determination that the first difference is greater than the second difference, and updating the power data packet further includes maintaining the system power level of the power data packet.

10. The method of claim 6, wherein the current power level of the first storage device is set to the sum of the first power level and the second difference based on the determination that the first difference is greater than the second difference.

11. The method of claim 10, further comprising: Replace the first power level of the first storage device with the current power level, wherein updating the power data packet includes increasing the system power level of the power data packet to the predefined power budget.

12. The method of claim 1, wherein the storage device ring further comprises a host device, a second storage device, and a third storage device, and the host device is coupled downstream of the second storage device and upstream of the third storage device in the power control path, and one or more programs further comprise instructions for performing the following operations: At the host device, the system power level and the power level of the plurality of storage devices are monitored.

13. The method of claim 2, further comprising: Based on the determination that the power insufficiency condition is met, the predefined power budget is increased to the updated power budget, wherein the system power level is configured to change below the predefined power budget increased to the updated power budget.

14. The method of claim 13, further comprising one of the following: The host module, self-coupled to one of the plurality of storage devices, obtains the updated power budget; and The predefined power budget is automatically increased by the predefined budget increment to generate the updated power budget.

15. The method of claim 13, wherein the power deficiency condition includes at least one of the following: The system power level has been maintained at the predefined power budget threshold duration; At least one predefined portion of the plurality of storage devices failed to increase the corresponding power level during the cycle of transmitting the power data packet; and The first storage device failed to increase the first power level within the first number of cycles.

16. The method of claim 1, wherein the electronic system includes a set of storage devices, the method further comprising: Select a subset of the group of storage devices of the electronic system as the plurality of storage devices; as well as Arranging the plurality of storage devices into the storage device ring includes assigning the upstream storage device and the downstream storage device to the first storage device.

17. The method of claim 1, wherein each of the plurality of storage devices operates in a power range having a power upper limit, and the plurality of storage devices have a predefined power budget that is less than a predefined portion of the sum of the power upper limits of all the plurality of storage devices.

18. The method of claim 1, further comprising: Before transmitting the power data packet along the power control path, at the initiating storage device: the system power level is set to a predefined power budget; and the power levels of multiple devices in the plurality of storage devices are set to be equal to each other, the sum of the power levels of the plurality of devices being equal to the predefined power budget.

19. A storage system comprising: One or more processors; as well as A memory having instructions stored thereon, which, when executed by the one or more processors, cause the processors to perform the method according to any one of claims 1 to 18.

20. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 18.

Images