Hybrid ssd space management method and device based on multi-agent reinforcement learning

Abstract

The application relates to the technical field of hybrid solid state disks, and discloses a hybrid SSD space management method and device based on multi-agent reinforcement learning, which models the coupled garbage collection and flash mode conversion operation in the hybrid SSD as a reinforcement learning problem decided by independent agents. A multi-agent reinforcement learning scheduler is added, the shared system state such as the SLC space proportion and the write frequency is monitored in real time, two agents independently select optimal actions from an action space containing cross-region recovery and mode degradation options according to respective strategies, and the optimal actions are combined into a joint command for execution, and finally, a differentiated reward function considering performance indexes and internal write overhead is used to dynamically adjust the scheduling strategy. The application solves the problem that garbage collection and mode conversion, two background operations, are independently executed, seriously compete for shared resources, easily interfere with each other, and result in the problems of I / O performance reduction and write delay sharp increase in the front end.

Description

Technical Field

[0001] This application relates to the field of hybrid solid-state drive technology, such as a hybrid SSD space management method and apparatus based on multi-agent reinforcement learning. Background Technology

[0002] With the continuous development of storage technology, solid-state drives (SSDs) have been widely used in data centers and consumer markets due to their high performance and low latency. Meanwhile, various types of flash memory storage cells, such as single-level cells (SLC), multi-level cells (MLC), triple-level cells (TLC), and quad-level cells (QLC), meet diverse scenarios with different performance and capacity requirements. SLC (single-level cell) is known for its high performance and durability, but its capacity density is relatively low; while QLC (quad-level cell) offers higher capacity density, but sacrifices performance and durability. As the demands for storage capacity and performance continue to increase, hybrid SSDs are gradually emerging to balance performance and capacity.

[0003] Hybrid SSDs allow flash memory cells to switch between different flash memory modes to achieve a balance between high performance and high density. Hybrid SSD designs are mainly divided into two categories: one is a fixed-ratio SLC and QLC hybrid SSD, and the other is a hybrid SSD that supports dynamic switching between SLC and QLC modes. In a fixed-ratio hybrid SSD, the SLC portion is typically used to cache hot data to improve performance, while the QLC portion is used to store cold data. However, this design requires precise control of the SLC and QLC ratio; otherwise, it may lead to wasted capacity or performance degradation. Hybrid SSDs that support dynamic switching, such as the Intel 665P and Micron Crucial P1, offer greater flexibility, but performance drops sharply when writing data to QLC mode after the SLC mode is exhausted, and the SLC-to-QLC conversion also introduces significant overhead.

[0004] In hybrid SSDs, garbage collection (GC) and flash cell mode conversion (MC) are two key operations. GC is used to clean up invalid data to free up space, while MC is used to convert between different flash cell types.

[0005] Garbage collection: Flash memory updates use an off-site update method, which means marking the original data as invalid and updating the data in a new space. The garbage collection mechanism is used to clean up this invalid data in a timely manner to free up more flash memory space. Before erasing the target block, garbage collection needs to migrate the valid data in the victim block to a safe space. Currently, most devices use a greedy garbage collection strategy for performance reasons, that is, selecting the block with the most invalid pages for erasure.

[0006] Mode switching: Hybrid SSDs support multiple operating modes on the same physical flash memory block by configuring the number of threshold voltage regions used in the block. This allows the block to be used as a low-bit-density block (for higher speed and reliability) or a high-bit-density block (for higher capacity). Mode switching is the mechanism that switches a block from one mode to another at runtime. In practice, switching is usually managed on a block-by-block basis, requiring the controller to first exhaust or migrate valid pages in the target block to safe space, then reinitialize the block in the new mode, and apply the appropriate page organization and programming rules. Many controllers use threshold-based switching strategies for simplicity; for example, expanding the low-bit pool when the device observes increased latency or free space pressure, and shrinking the low-bit pool when capacity pressure dominates.

[0007] However, in existing designs, in hybrid SSDs that support dynamic SLC / QLC conversion, the two background operations of garbage collection and mode switching are executed independently and compete for shared resources, which can easily interfere with each other, resulting in a bottleneck of a sharp drop in front-end input / output (I / O) performance and a sharp increase in write latency.

[0008] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application. Summary of the Invention

[0009] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0010] This disclosure provides a hybrid SSD space management method and apparatus based on multi-agent reinforcement learning, which solves the bottleneck in hybrid SSDs that support dynamic SLC / QLC conversion. The two background operations, garbage collection and mode conversion, are executed independently and compete for shared resources, which are prone to mutual interference, resulting in a sharp drop in front-end I / O performance and a sharp increase in write latency.

[0011] According to a first aspect of this disclosure, a hybrid SSD space management method based on multi-agent reinforcement learning is provided, the method comprising: During the operation of the hybrid solid-state drive (SSD), observe the shared status of the system. The shared state is input into the waste recycling agent and the mode conversion agent, respectively. The waste recycling agent determines the first target action from a preset waste recycling action space based on a first reinforcement learning strategy. The mode-switching agent determines the second target action from a preset mode-switching action space based on a second reinforcement learning strategy. Combine the first target action and the second target action into a joint control command; Execute the joint control command to perform the corresponding garbage collection operation and / or flash memory mode conversion operation; The reward values ​​for the garbage collection agent and the mode switching agent are calculated based on the execution results, and the corresponding reinforcement learning policies are updated using the reward values ​​to achieve coordinated scheduling of garbage collection and mode switching operations.

[0012] In some implementations of the first aspect, the aforementioned shared state includes: SLC space ratio, which represents the proportion of the total number of SLC blocks to the total space; MLC space ratio, which represents the proportion of the total number of MLC blocks to the total space; space utilization, which represents the ratio of the size of written data to the total capacity; write frequency, which represents the intensity of data writing; and previously adopted action, which represents the joint control command executed in the previous scheduling cycle.

[0013] In some implementations of the first aspect, the aforementioned garbage collection action space includes: an empty action, used to indicate that no garbage collection operation is performed; an intra-regional collection action, used to perform garbage collection within the same flash memory mode region; and an inter-regional collection action, used to migrate valid data from low bit density regions to high bit density regions during garbage collection to increase storage capacity.

[0014] In some implementations of the first aspect, the aforementioned regional recycling action includes at least one of SLC regional recycling, MLC regional recycling, and QLC regional recycling; the cross-regional recycling action includes at least one of SLC to MLC data migration recycling and MLC to QLC data migration recycling.

[0015] In some implementations of the first aspect, the aforementioned mode conversion action space includes: an empty action, used to indicate that no mode conversion operation is performed; and a mode degradation action, used to convert a flash memory block from a low bit density mode to a high bit density mode in order to restore or increase the storage capacity density.

[0016] In some implementations of the first aspect, the aforementioned mode degradation action includes at least one of the mode conversion from SLC to MLC and the mode conversion from MLC to QLC.

[0017] In some implementations of the first aspect, the reward value of the aforementioned garbage collection agent is determined based on the performance index score observed during the scheduling cycle, the amount of internal data written introduced by the garbage collection operation, and the change in the proportion of available blocks in the fast region after the operation is executed. The performance metric score is calculated based on the average request latency and average throughput within the period.

[0018] In some implementations of the first aspect, the reward value of the aforementioned mode-switching agent is determined based on the front-end performance index score observed during the scheduling cycle, the amount of internal data written introduced by the mode-switching operation, and the change in the global available storage capacity after the operation is executed. The performance metric score is calculated based on the average request latency and average throughput within the period.

[0019] In some implementations of the first aspect, both the first reinforcement learning strategy and the second reinforcement learning strategy are based on the Q-learning algorithm. The waste recycling agent employs a smaller discount factor and a larger exploration rate than the mode switching agent, so that waste recycling decisions focus on short-term dynamic responses.

[0020] According to a second aspect of this disclosure, a hybrid SSD space management apparatus based on multi-agent reinforcement learning is provided for performing the method of the first aspect; the apparatus includes: The observation module is used to monitor the shared state of the system during the operation of the hybrid solid-state drive (SSD). The processing module is used to input the shared state to the waste recycling agent and the mode conversion agent respectively; The processing module is also used for the waste recycling agent to determine a first target action from a preset waste recycling action space based on a first reinforcement learning strategy. The processing module is also used for the mode-switching agent to determine a second target action from a preset mode-switching action space based on a second reinforcement learning strategy. The processing module is further configured to combine the first target action and the second target action into a joint control command; The execution module is used to execute the joint control commands to perform corresponding garbage collection operations and / or flash memory mode conversion operations; The execution module is further configured to calculate the reward values ​​of the garbage collection agent and the mode switching agent based on the execution results, and update the corresponding reinforcement learning policies using the reward values ​​to achieve coordinated scheduling of garbage collection and mode switching operations.

[0021] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the program to implement the method of the first aspect.

[0022] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect.

[0023] The hybrid SSD space management method, apparatus, device, and storage medium based on multi-agent reinforcement learning provided in this disclosure can achieve the following technical effects: This disclosure provides a hybrid SSD space management method, apparatus, device, and storage medium based on multi-agent reinforcement learning. It models the tightly coupled background operations of garbage collection (GC) and flash mode conversion (MC) in hybrid SSDs as a reinforcement learning problem decided by independent agents. The solution adds a multi-agent reinforcement learning scheduler containing a garbage collection agent and a mode conversion agent. By monitoring the shared system status in real time, such as the SLC space ratio and write frequency, the two agents independently select the optimal action from an action space containing options such as cross-region garbage collection and mode degradation, based on their respective Q-learning strategies. These actions are then combined into a joint command for execution. Finally, the scheduling strategy is dynamically adjusted based on a differentiated reward function that balances front-end performance metrics and internal write overhead. Therefore, this invention effectively solves the bottleneck of performance degradation caused by independent execution and mutual interference of GC and MC in existing technologies. It achieves intelligent collaboration between the two types of background operations, significantly reducing the interference of long-latency operations on front-end I / O requests while ensuring the safety of free space, thereby improving the overall storage performance and service quality of hybrid SSDs.

[0024] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0025] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a flowchart illustrating a hybrid SSD space management method based on multi-agent reinforcement learning provided in an embodiment of this disclosure. Figure 2 This is an architecture diagram of a hybrid SSD space management system based on multi-agent reinforcement learning provided in an embodiment of this disclosure; Figure 3 This is a spatial scheduling framework diagram based on multi-agent reinforcement learning provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a hybrid SSD space management device based on multi-agent reinforcement learning provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of a hybrid SSD space management device based on multi-agent reinforcement learning, provided in an embodiment of this disclosure. Detailed Implementation

[0026] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0027] The terms "first," "second," etc., used in the embodiments of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0028] Unless otherwise stated, the term "multiple" means two or more.

[0029] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0030] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0031] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0032] To address the aforementioned issues, this disclosure provides a hybrid SSD space management method, apparatus, device, and storage medium based on multi-agent reinforcement learning.

[0033] The following description, in conjunction with the accompanying drawings, illustrates the hybrid SSD space management method, apparatus, device, and storage medium based on multi-agent reinforcement learning provided in the embodiments of this disclosure.

[0034] Figure 1 This is a flowchart illustrating a hybrid SSD space management method based on multi-agent reinforcement learning provided in an embodiment of this disclosure.

[0035] like Figure 1 As shown, a hybrid SSD space management method based on multi-agent reinforcement learning is proposed, which includes: S101, during the operation of the hybrid solid-state drive (SSD), observes the shared status of the system; S102, input the shared state to the waste collection agent and the mode conversion agent respectively; S103, the waste recycling agent decides on the first target action from the preset waste recycling action space based on the first reinforcement learning strategy; S104, the mode-switching agent makes a decision on the second target action from the preset mode-switching action space based on the second reinforcement learning strategy; S105, combine the first target action and the second target action into a joint control command; S106, Execute joint control commands to perform corresponding garbage collection operations and / or flash memory mode conversion operations; S107, calculate the reward values ​​of the garbage collection agent and the mode switching agent according to the execution results, and use the reward values ​​to update the corresponding reinforcement learning policies to achieve coordinated scheduling of garbage collection and mode switching operations.

[0036] In some embodiments, the shared state includes SLC space ratio, which represents the proportion of the total number of SLC blocks to the total space; MLC space ratio, which represents the proportion of the total number of MLC blocks to the total space; space utilization, which represents the ratio of the size of written data to the total capacity; write frequency, which represents the intensity of data writing; and previously adopted action, which represents the joint control command executed in the previous scheduling cycle.

[0037] In some embodiments, the above-mentioned garbage collection action space includes: an empty action, which indicates that no garbage collection operation is performed; an intra-regional collection action, which is used to perform garbage collection within the same flash memory mode region; and an inter-regional collection action, which is used to migrate valid data from a low bit density region to a high bit density region during garbage collection to increase storage capacity.

[0038] In some embodiments, the above-mentioned intra-regional recycling action includes at least one of SLC intra-regional recycling, MLC intra-regional recycling, and QLC intra-regional recycling; the cross-regional recycling action includes at least one of SLC to MLC data migration recycling and MLC to QLC data migration recycling.

[0039] In some embodiments, the above-mentioned mode conversion action space includes: an empty action, which indicates that no mode conversion operation is performed; and a mode degradation action, which is used to convert the flash memory block from a low bit density mode to a high bit density mode in order to restore or increase the storage capacity density.

[0040] In some embodiments, the mode degradation action described above includes at least one of mode conversion from SLC to MLC and mode conversion from MLC to QLC.

[0041] In some embodiments, the reward value of the above-mentioned garbage collection agent is determined based on the performance index score observed during the scheduling cycle, the amount of internal data written introduced by the garbage collection operation, and the change in the proportion of available blocks in the fast region after the operation is performed. The performance score is calculated based on the average request latency and average throughput within the period.

[0042] In some embodiments, the reward value of the mode-switching agent is determined based on the front-end performance index score observed during the scheduling period, the amount of internal data written introduced by the mode-switching operation, and the change in the global available storage capacity after the operation is performed. The performance score is calculated based on the average request latency and average throughput within the period.

[0043] In some embodiments, both the first reinforcement learning strategy and the second reinforcement learning strategy described above are implemented based on the Q-learning algorithm; Among them, the waste recycling agent uses a smaller discount factor and a larger exploration rate than the mode switching agent, so that waste recycling decisions focus on short-term dynamic responses.

[0044] Figure 2 This is an architecture diagram of a hybrid SSD space management system based on multi-agent reinforcement learning provided in an embodiment of this disclosure. Figure 3 This is a spatial scheduling framework diagram based on multi-agent reinforcement learning provided in this disclosure embodiment, combined with... Figure 2 and Figure 3 ,right Figure 1 The hybrid SSD space management method based on multi-agent reinforcement learning is further described in this paper.

[0045] Hybrid SSD space management involves tightly coupled background activities, including garbage collection and mode switching. Garbage collection reclaims invalid pages to maintain a sufficient supply of free space, while mode switching reshapes the capacity and performance balance by altering the effective bit density of blocks. These activities operate under shared constraints, such as free blocks and internal write bandwidth. The main challenge lies in the fact that, under these shared resources, garbage collection and mode switching are guided by different but tightly coupled signals. Garbage collection focuses on the efficiency of reclamation within each mode region, which largely depends on the proportion of invalid pages and the resulting cost of copying valid pages. Mode switching focuses on device-level space pressure and the proportion of the current mode region, as it alters the effective capacity density by reallocating blocks across modes. However, decisions to improve one signal often interfere with the other, making it difficult to simultaneously maintain free space security and stable foreground performance.

[0046] The overall architecture of the hybrid SSD space management scheme based on multi-agent reinforcement learning includes six main functional modules: address translation, wear leveling, data heat classification, translation controller, garbage collection, and reinforcement learning-assisted GC-MC scheduler.

[0047] The address translation module maps logical addresses to physical addresses in the flash memory, ensuring correct data storage and retrieval. The wear leveling module extends SSD lifespan by balancing the program / erase (P / E) cycles of flash memory cells, ensuring all cells experience roughly the same level of wear and preventing premature failure. The data hotness classification module uses the K-Means algorithm to categorize data into hot and cold data, storing them in separate flash blocks to reduce effective data copying and write amplification during garbage collection. The conversion controller manages the conversion process between flash memory cells, dynamically adjusting the proportion of different levels of flash memory cells based on system status and workload requirements to optimize performance and capacity utilization. The garbage collection module cleans up invalid data, freeing up more usable space. Garbage collection is performed individually on each flash memory cell, prioritizing the block containing the most invalid pages for erasure and data migration.

[0048] The multi-agent reinforcement learning-assisted GC-MC scheduler is the core module of this solution. It intelligently decides when and how to perform garbage collection (GC) and flash mode conversion (MC) operations by monitoring the internal state and workload patterns of the SSD in real time. Based on an online lightweight Q-learning model, this scheduler can dynamically adjust its strategy according to environmental feedback to optimize system performance.

[0049] To address this challenge, this solution formulates the scheduling problem as a multi-agent reinforcement learning problem. Specifically, the solution employs two agents that make independent decisions within a shared system state: the first agent is responsible for operations related to garbage collection, while the second agent focuses on mode transition operations. At each decision step t, the environment is represented by the shared state. The two agents choose their respective basic actions. and The controller combines them into a single command: , Joint commands originate from the joint action space. It is then executed to perform coupling control operations. Q-learning is employed, where each agent maintains its own Q-table. It learns local policies based on shared state. The key design components described above are described in detail below.

[0050] Shared State: The state representation of garbage collection agents and mode-switching agents should be able to gather sufficient information from the environment and characterize historical situations so that the agents can learn and make good decisions. Therefore, the state representation should consider three aspects: device state, workload patterns, and previous actions.

[0051] Specifically, the settings are as follows: SLC space ratio: This indicates the proportion of the total number of blocks in the SLC region to the total space under the current conditions, and also reflects the maximum capacity that can be provided under the current conditions.

[0052] MLC space ratio: This indicates the proportion of the total number of blocks in the MLC region to the total space under the current conditions, and also reflects the maximum capacity that can be provided under the current conditions.

[0053] Space utilization: This represents the ratio of the amount of data written to the total capacity of the solid-state drive.

[0054] Write frequency: Indicates the intensity of data writing in the solid-state drive.

[0055] Previous action: Indicates the action taken in the previous request cycle.

[0056] Therefore, the total number of states is the product of the number of choices for the five state representations mentioned above.

[0057] The garbage collection agent. This agent determines whether to trigger garbage collection and where to perform it.

[0058] Actions. The action space of the waste recycling agent is: , Within a given region, actions reclaim space within the same region. Cross-region actions migrate valid pages to a higher-level region during reclamation to increase SSD capacity. Notably, the solution includes a null action (NULL) as an option, which performs no operation based on the current environment.

[0059] Rewards. The reward function guides the agent to learn optimal actions based on the environment; higher rewards correspond to shorter response times, and vice versa. The garbage collection agent in hybrid SSDs has a clear objective: to minimize data movement in the reclaimed space and, where possible, release more blocks in faster regions to maintain performance stability. To achieve this objective, this scheme calculates a periodic reward by considering latency and throughput within the scheduling cycle, internal write bytes introduced by garbage collection, and changes in available blocks in fast regions (such as SLC and MLC). The reward is defined as: , in, This represents the normalized internal write bytes caused by garbage collection within period t. This indicates the change in the proportion of available blocks in the fast region (SLC region and MLC region) within the same period. The larger the value, the greater the effective page migration overhead. A positive value indicates that waste recycling has successfully increased the capacity of the fast zone by adding collection blocks to the fast zone. Based on periodic indicators (Derived from latency and throughput), the performance observed within period t is summarized using piecewise scores. Let... , and Configure the typical latency range of different mode regions (such as SLC, MLC, and QLC) under typical load conditions, then , in, It is the average latency of all requests observed within period t. It is the normalized average throughput within the same period.

[0060] The solution calculates the latency by averaging the latency of all requests completed within the period. : , in, It is the number of requests completed within period t. This is the latency of the i-th request. Average throughput is calculated by dividing the number of front-end bytes completed by the cycle duration. and normalize it to This is to make the scale comparable across different workloads and device configurations.

[0061] The mode-switching agent determines whether to perform a mode switch and the direction of the mode-switching block across mode regions.

[0062] Actions. The action space of the mode-switching agent is: , The action space converts a block from low-bit mode to high-bit mode to restore capacity density when space pressure increases. Similarly, the action space also includes a null action (NULL), which performs no conversions when the current region composition is sufficient or conversion overhead should be avoided.

[0063] Rewards. The goal of the mode-switching agent complements that of the garbage-collecting agent; it should adjust the region composition to maintain sufficient overall free capacity while avoiding excessive switching-induced data movement (which would degrade foreground performance). To achieve this, the scheme calculates a periodic reward by considering latency and throughput within the scheduling cycle, internal write bytes introduced by mode switching, and the overall capacity of the solid-state drive. This reward is defined as: , in, This represents the normalized internal write bytes caused by mode switching within period t. This indicates the normalized available solid-state drive capacity within the same period. The larger the value, the higher the conversion overhead due to block reconstruction and data migration. A larger value indicates a better global spatial condition, reducing the possibility of spatial collapse. This scheme calculates the value based on the total number of available free blocks across all mode regions. This is then normalized to make the scale comparable across different workloads and device configurations. The term encourages mode-switching agents to trigger transitions as early as possible to prevent global space pressure, while learning to avoid unnecessary transitions (which introduce excessive internal writes and interfere with foreground I / O). Shared with the waste recycling agent to ensure consistent optimization goals, and based on the average request latency within period t. and normalized average throughput .

[0064] By adjusting the hyperparameters of each agent, learning stability and adaptation speed can be balanced. Since the garbage collection agent is invoked more frequently and directly impacts foreground latency through data migration, a relatively small discount factor and a high exploration rate are used to encourage responsive decisions under short-term dynamics. In contrast, the mode-switching agent is triggered less frequently and primarily adjusts the long-term region composition; therefore, a larger discount factor and a smaller exploration rate are used to emphasize long-term rewards and avoid unnecessary transitions.

[0065] A multi-agent scheduler is periodically triggered to coordinate garbage collection and mode switching. At the end of a scheduling cycle, the controller first observes the shared system state, then allows the garbage collection agent and the mode switching agent to independently select their actions using Q-learning. The selected actions are combined into a joint command, executing the corresponding background management operation. After the command is executed, the controller observes the next state, calculates agent-specific rewards based on the reward definition, and updates the two Q-tables accordingly.

[0066] To make online learning feasible in the storage system, a warm-start approach is adopted for the Q-table through an offline pre-training phase. This approach pre-trains the agent using a representative sample workload to obtain a conservative initial policy. The reason for offline training is that certain actions in hybrid SSDs are often shared by all applications. For example, when the drive is nearly empty, there is no need to trigger any garbage collection or mode switching. Therefore, using an untrained reinforcement learning model (without pre-training), unnecessary actions would degrade the performance of the hybrid SSD and shorten its lifespan. Furthermore, untrained reinforcement learning models require warm-up time and may result in poor performance. After offline training, the reinforcement learning model will be tuned and used in an online training manner to adapt to new incoming applications.

[0067] In a specific example, the following section provides a detailed explanation of the proposed hybrid SSD space management method based on multi-agent reinforcement learning, namely the GC-MC scheduling strategy based on multi-agent reinforcement learning, using algorithm pseudocode. The detailed implementation process of this invention is given below:

[0068] This invention proposes a hybrid SSD space management method based on multi-agent reinforcement learning. It models the tightly coupled background operations of garbage collection (GC) and flash mode conversion (MC) in hybrid SSDs as a reinforcement learning problem decided by independent agents. The scheme adds a multi-agent reinforcement learning scheduler containing a garbage collection agent and a mode conversion agent. By monitoring the shared system status in real time, such as the SLC space ratio and write frequency, the two agents independently select the optimal action from an action space containing options such as cross-region garbage collection and mode degradation, based on their respective Q-learning strategies. These actions are then combined into a joint command for execution. Finally, the scheduling strategy is dynamically adjusted based on a differentiated reward function that balances front-end performance metrics and internal write overhead. Therefore, this invention effectively solves the bottleneck of performance degradation caused by independent execution and mutual interference of GC and MC in existing technologies. It achieves intelligent collaboration between the two types of background operations, significantly reducing the interference of long-latency operations on front-end I / O requests while ensuring the safety of free space, thereby improving the overall storage performance and service quality of hybrid SSDs. This effectively improves the data management efficiency of hybrid SSDs, increases the overall system throughput, greatly reduces write amplification issues during the process, and ensures the efficient operation of SSDs.

[0069] and Figure 1 Corresponding to the hybrid SSD space management method based on multi-agent reinforcement learning disclosed in the previous disclosure, this disclosure also provides a hybrid SSD space management device based on multi-agent reinforcement learning, such as... Figure 4 As shown, a hybrid SSD space management device based on multi-agent reinforcement learning may include: Observation module 401 is used to observe the shared status of the system during the operation of the hybrid solid-state drive (SSD); Processing module 402 is used to input the shared state to the waste collection agent and the mode conversion agent respectively; The processing module 402 is also used to enable the waste recycling agent to determine the first target action from the preset waste recycling action space based on the first reinforcement learning strategy. The processing module 402 is also used to enable the mode-switching agent to determine a second target action from a preset mode-switching action space based on a second reinforcement learning strategy. Processing module 402 is also used to combine the first target action and the second target action into a joint control command; Execution module 403 is used to execute joint control commands to perform corresponding garbage collection operations and / or flash memory mode conversion operations; The execution module 403 is also used to calculate the reward values ​​of the garbage collection agent and the mode switching agent according to the execution results, and use the reward values ​​to update the corresponding reinforcement learning policies to achieve coordinated scheduling of garbage collection and mode switching operations.

[0070] In some embodiments, the shared state includes SLC space ratio, which represents the proportion of the total number of SLC blocks to the total space; MLC space ratio, which represents the proportion of the total number of MLC blocks to the total space; space utilization, which represents the ratio of the size of written data to the total capacity; write frequency, which represents the intensity of data writing; and previously adopted action, which represents the joint control command executed in the previous scheduling cycle.

[0071] In some embodiments, the above-mentioned garbage collection action space includes: an empty action, which indicates that no garbage collection operation is performed; an intra-regional collection action, which is used to perform garbage collection within the same flash memory mode region; and an inter-regional collection action, which is used to migrate valid data from a low bit density region to a high bit density region during garbage collection to increase storage capacity.

[0072] In some embodiments, the above-mentioned intra-regional recycling action includes at least one of SLC intra-regional recycling, MLC intra-regional recycling, and QLC intra-regional recycling; the cross-regional recycling action includes at least one of SLC to MLC data migration recycling and MLC to QLC data migration recycling.

[0073] In some embodiments, the above-mentioned mode conversion action space includes: an empty action, which indicates that no mode conversion operation is performed; and a mode degradation action, which is used to convert the flash memory block from a low bit density mode to a high bit density mode in order to restore or increase the storage capacity density.

[0074] In some embodiments, the aforementioned mode degradation action includes at least one of SLC to MLC mode conversion and MLC to QLC mode conversion.

[0075] In some embodiments, the reward value of the above-mentioned garbage collection agent is determined based on the performance index score observed during the scheduling cycle, the amount of internal data written introduced by the garbage collection operation, and the change in the proportion of available blocks in the fast region after the operation is performed. The performance metric score is calculated based on the average request latency and average throughput within the period.

[0076] In some embodiments, the reward value of the mode-switching agent is determined based on the front-end performance index score observed during the scheduling period, the amount of internal data written introduced by the mode-switching operation, and the change in the global available storage capacity after the operation is performed. The performance metric score is calculated based on the average request latency and average throughput within the period.

[0077] In some embodiments, both the first reinforcement learning strategy and the second reinforcement learning strategy described above are implemented based on the Q-learning algorithm; Among them, the waste recycling agent uses a smaller discount factor and a larger exploration rate than the mode switching agent, so that waste recycling decisions focus on short-term dynamic responses.

[0078] Combination Figure 5 As shown in the illustration, this disclosure also provides a hybrid SSD space management device 500 based on multi-agent reinforcement learning, including a processor 504 and a memory 501. Optionally, the system may further include a communication interface 502 and a bus 503. The processor 504, communication interface 502, and memory 501 can communicate with each other via the bus 503. The communication interface 502 can be used for information transmission. The processor 504 can call logical instructions in the memory 501 to execute the hybrid SSD space management method based on multi-agent reinforcement learning described in the above embodiments.

[0079] Furthermore, the logic instructions in the aforementioned memory 501 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0080] The memory 501, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 504 executes functional applications and data processing by running the program instructions / modules stored in the memory 501, thereby implementing the hybrid SSD space management method based on multi-agent reinforcement learning in the above embodiments.

[0081] The memory 501 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 501 may include high-speed random access memory and may also include non-volatile memory.

[0082] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to execute a hybrid SSD space management method based on multi-agent reinforcement learning.

[0083] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0084] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0085] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0086] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0087] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0088] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0089] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0090] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0091] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A hybrid SSD space management method based on multi-agent reinforcement learning, characterized in that, The method includes: During the operation of the hybrid solid-state drive (SSD), observe the shared status of the system. The shared state is input into the waste recycling agent and the mode conversion agent, respectively. The waste recycling agent determines the first target action from a preset waste recycling action space based on a first reinforcement learning strategy. The mode-switching agent determines the second target action from a preset mode-switching action space based on a second reinforcement learning strategy. Combine the first target action and the second target action into a joint control command; Execute the joint control command to perform the corresponding garbage collection operation and / or flash memory mode conversion operation; The reward values ​​for the garbage collection agent and the mode switching agent are calculated based on the execution results, and the corresponding reinforcement learning policies are updated using the reward values ​​to achieve coordinated scheduling of garbage collection and mode switching operations.

2. The method according to claim 1, characterized in that, The shared state includes SLC space ratio, which represents the proportion of the total number of SLC blocks to the total space; MLC space ratio, which represents the proportion of the total number of MLC blocks to the total space; space utilization, which represents the ratio of the size of written data to the total capacity; write frequency, which represents the intensity of data writing; and previously adopted action, which represents the joint control command executed in the previous scheduling cycle.

3. The method according to claim 1, characterized in that, The garbage collection action space includes: an empty action, which indicates that no garbage collection operation is performed; an intra-regional garbage collection action, which is used to perform garbage collection within the same flash memory mode region; and an inter-regional garbage collection action, which is used to migrate valid data from low bit density regions to high bit density regions during garbage collection to increase storage capacity.

4. The method according to claim 3, characterized in that, The intra-regional recycling action includes at least one of SLC intra-regional recycling, MLC intra-regional recycling, and QLC intra-regional recycling; the cross-regional recycling action includes at least one of SLC to MLC data migration recycling and MLC to QLC data migration recycling.

5. The method according to claim 1, characterized in that, The mode conversion action space includes: an empty action, which indicates that no mode conversion operation is performed; and a mode degradation action, which is used to convert the flash memory block from a low bit density mode to a high bit density mode in order to restore or increase the storage capacity density.

6. The method according to claim 5, characterized in that, The mode downgrade action includes at least one of the following: mode conversion from SLC to MLC and mode conversion from MLC to QLC.

7. The method according to claim 1, characterized in that, The reward value of the waste recycling agent is determined based on the performance index score observed during the scheduling cycle, the amount of internal data written introduced by the waste recycling operation, and the change in the proportion of available blocks in the fast region after the operation is executed. The performance metric score is calculated based on the average request latency and average throughput within the period.

8. The method according to claim 1, characterized in that, The reward value of the mode-switching agent is determined based on the front-end performance index score observed during the scheduling cycle, the amount of internal data written introduced by the mode-switching operation, and the change in the global available storage capacity after the operation is executed. The performance metric score is calculated based on the average request latency and average throughput within the period.

9. The method according to claim 1, characterized in that, Both the first reinforcement learning strategy and the second reinforcement learning strategy are implemented based on the Q-learning algorithm; The waste recycling agent employs a smaller discount factor and a larger exploration rate than the mode switching agent, so that waste recycling decisions focus on short-term dynamic responses.

10. A hybrid SSD space management device based on multi-agent reinforcement learning, characterized in that, The device includes: The observation module is used to monitor the shared state of the system during the operation of the hybrid solid-state drive (SSD). The processing module is used to input the shared state to the waste recycling agent and the mode conversion agent respectively; The processing module is also used for the waste recycling agent to determine a first target action from a preset waste recycling action space based on a first reinforcement learning strategy. The processing module is also used for the mode-switching agent to determine a second target action from a preset mode-switching action space based on a second reinforcement learning strategy. The processing module is further configured to combine the first target action and the second target action into a joint control command; The execution module is used to execute the joint control commands to perform corresponding garbage collection operations and / or flash memory mode conversion operations; The execution module is further configured to calculate the reward values ​​of the garbage collection agent and the mode switching agent based on the execution results, and update the corresponding reinforcement learning policies using the reward values ​​to achieve coordinated scheduling of garbage collection and mode switching operations.

Images