Universal middleware system built based on ZNS SSD and related method

By designing a general middleware system based on ZNS SSD, the problems of limited performance improvement and high cost of traditional flash SSDs are solved, and compatibility and storage performance optimization of ZNS SSD are achieved. Through adapting sequential writing characteristics and garbage collection mechanism, Ensures compatibility and performance of the storage system.

CN119045753AActive Publication Date: 2024-11-29UNIV OF SCI & TECH OF CHINA

Patent Information

Application Number
CN202411556141.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-04
Publication Date
2024-11-29
Estimated Expiration
2044-11-04

AI Technical Summary

Technical Problem

Traditional flash solid-state drives cannot achieve hardware and software collaborative design, resulting in limited performance improvement and high cost. ZNS SSD cannot be compatible with existing storage system stacks due to sequential write characteristics.

Method used

Design a general middleware system based on ZNS SSD, including target component module, creation component module, address mapping module, Zone management module, garbage collection module and data recovery module, and adapt to ZNS by intercepting I/O requests from the upper-level system. SSD writes the sequential features and realizes compatibility with ZNS SSD without changing the upper layer application.

Benefits of technology

It realizes that ZNS SSD is compatible with existing storage system stack without sacrificing performance, while saving costs, and optimizes storage space utilization through garbage collection mechanisms to ensure storage performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119045753A_ABST
    Figure CN119045753A_ABST
Patent Text Reader

Abstract

The invention discloses a general middleware system built based on a ZNS SSD and a related method, and the general middleware system is a corresponding scheme, and in the scheme, after a request downwards sent by an upper layer system is intercepted, according to an address mapping relation, random writing of an application logic address is converted into sequential writing of a physical address, and applied data is stored in a correct position in the ZNS SSD; moreover, when the space is insufficient, the Zone with the most invalid data can be sorted and recycled through garbage collection, so that space is vacated for subsequent new data writing while the storage performance is ensured; besides, the scheme can provide a universal block device interface for an upper-layer system without making any change on the existing upper-layer system or sacrificing performance, so that the ZNS SSD can be compatible with the existing storage system stack.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, and in particular to a general middleware system based on ZNS SSD and related methods. Background Art

[0002] In recent years, flash solid-state drives (Flash SSDs) have been widely studied and applied in both industry and academia. Since flash solid-state drives have lower latency and higher performance than traditional hard disks (HDDs), they are widely used in different scenarios such as cloud service storage vendors, high-performance computing centers, and personal computers. Flash solid-state drives consist of four parts: host interface logic, solid-state drive controller, dynamic random access memory, and a multi-channel subsystem that connects multiple NAND chips (flash memory chips), providing a block device interface to the upper layer.

[0003] Traditional flash solid-state drives have the following problems: their data management is fully managed by the hard drive's internal controller and is invisible to the upper layer, so it is impossible to achieve software and hardware collaborative design to improve performance; and the cost of flash solid-state drives is high due to their large demand for expensive dynamic random access memory and the need to reserve additional NAND chip space for hard drive data transfer. In order to solve these problems, a new flash solid-state drive came into being. ZNS SSD (space solid-state drive) uses the new NVMe (logical device interface specification) interface protocol, provides a zone interface composed of multiple blocks to the upper layer, and provides a new set of hard drive control commands. ZNS SSD exposes the flash translation layer (FTL) to the upper layer, and leaves the hard drive data management to the upper layer, saving some dynamic random access memory overhead of the storage address mapping table, and does not require additional reserved space, saving costs, and providing opportunities for software and hardware collaboration.

[0004] However, since ZNS SSD has only been released recently, although there are various developments for using ZNS SSD, due to its special restrictions on sequential write zone interface, ZNS SSD is not compatible with the existing storage system stack. Therefore, it is necessary to develop a general middleware based on ZNS SSD.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a general middleware system and related methods built on ZNS SSD, which can provide a general block device interface to the upper layer, intercept the I / O request sent by the upper layer system, and adaptively modify the I / O request according to the characteristics of ZNS SSD without making any changes to the existing upper layer applications and without sacrificing performance, so that ZNS SSD can be compatible with the existing storage system stack.

[0007] The objective of the present invention is achieved through the following technical solutions: A general middleware system based on ZNS SSD includes: a target component module, a component creation module, an address mapping module, a zone management module, a garbage collection module and a data recovery module; wherein: Create component modules for initialization and resource allocation of common middleware systems; The target component module is used to intercept and process requests from the upper system, add metadata of the ZNS SSD to adapt to the sequential write characteristics of the ZNS SSD, call the Zone management module, determine the Zone for response, and call the address mapping module to map the block device logical address to the corresponding Zone logical address, and then send it to the ZNS SSD for data operation; wherein the ZNS SSD is a space solid state drive and the Zone is a partition; The address mapping module is used to maintain the dynamic mapping relationship between the block device logical address and the zone logical address in the ZNS SSD; Zone management module, used to monitor and manage the zone status of each partition in ZNS SSD, and select the zone for response; The garbage collection module is used to detect the proportion of invalid data in the zone and perform data migration and zone reset operations when the invalid data reaches the garbage collection threshold; The data recovery module is used to restore the metadata information of the ZNS SSD by reading the persistent log data after a system failure.

[0008] A method for implementing a general middleware system based on ZNS SSD, comprising: setting a creation component module for initialization and resource allocation of the general middleware system; setting a target component module for intercepting and processing requests from an upper system, adding metadata of the ZNS SSD to adapt to the sequential writing characteristics of the ZNS SSD, calling a zone management module, determining a zone for response, and calling an address mapping module to map a block device logical address to a corresponding zone logical address, and then sending it to the ZNS SSD for data operation; setting an address mapping module for maintaining a dynamic mapping relationship between a block device logical address and a zone logical address in the ZNS SSD; setting a zone management module for monitoring and managing the zone status of each partition in the ZNS SSD and selecting a zone for response; setting a garbage collection module for detecting the proportion of invalid data in the zone and performing data migration and zone reset operations when the invalid data reaches a garbage collection threshold; setting a target component module for recovering the metadata information of the ZNS SSD by reading persistent log data after a system failure; wherein the ZNS SSD is a space solid state drive and the Zone is a partition; Based on the target component module, the creation component module, the address mapping module, the Zone management module, the garbage collection module and the data recovery module, the aforementioned general middleware system based on ZNS SSD is constructed.

[0009] It can be seen from the technical solution provided by the present invention that after intercepting the request sent downward by the upper-level system, the random write of the application logical address is converted into the sequential write of the physical address according to the address mapping relationship, and the application data is stored in the correct position in the ZNS SSD; and when there is insufficient space, garbage collection can be used to sort out and recycle the Zone with the most invalid data, thereby making room for subsequent new data writing while ensuring storage performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0011] Figure 1 A schematic diagram of a general middleware system based on ZNS SSD provided in an embodiment of the present invention; Figure 2 A schematic diagram of the overall storage system architecture including the ZNS SSD provided in an embodiment of the present invention; Figure 3A schematic diagram of the middleware system architecture based on ZNS SSD provided in an embodiment of the present invention; Figure 4 A schematic diagram of Zone state management switching provided by an embodiment of the present invention; Figure 5 A schematic diagram of Zone partition management of a ZNS SSD provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0013] First, the terms that may be used in this article are explained as follows: The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0014] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0015] The following is a detailed description of a general middleware system based on ZNS SSD and related methods provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in this field. If no specific conditions are specified in the embodiments of the present invention, they shall be carried out in accordance with the conventional conditions in the field or the conditions recommended by the manufacturer. The instruments used in the embodiments of the present invention, for which the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0016] Embodiment 1 The embodiment of the present invention provides a general middleware system based on ZNS SSD, such as Figure 1 As shown in the figure, it mainly includes: target component module (DM-Target), component creation module (Init), address mapping module (Mapping Table), Zone management module (Zone Manager), garbage collection module (Garbage Collection) and data recovery module (Recovery).

[0017] (1) Create component modules for the initialization and resource allocation of the general middleware system.

[0018] (2) The target component module is used to intercept and process requests from the upper system (for example, the above-mentioned application or file system), add the metadata of the ZNS SSD to adapt to the sequential write characteristics of the ZNS SSD, call the Zone management module, determine the Zone for response, and call the address mapping module to map the block device logical address to the corresponding Zone logical address, and then send it to the ZNS SSD for data operation.

[0019] (3) Address mapping module, used to maintain the dynamic mapping relationship between the block device logical address and the zone logical address in the ZNS SSD.

[0020] (4) Zone management module, which is used to monitor and manage the zone status of each partition in the ZNS SSD and select the zone for response.

[0021] (5) Garbage collection module, which is used to detect the proportion of invalid data in the zone; when the invalid data reaches the garbage collection threshold, it performs data migration and zone reset operations.

[0022] (6) Data recovery module, used to restore the metadata information of ZNS SSD by reading the persistent log data after a system failure.

[0023] The above-mentioned system provided by the embodiment of the present invention is a middleware system that can be directly deployed on the ZNS SSD without modification to the upper-layer system. It intercepts the requests sent downward by the application and adaptively modifies the I / O requests according to the characteristics of the ZNS SSD, so that the ZNS SSD can be compatible with the existing storage system stack.

[0024] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the system provided by the embodiment of the present invention is described in detail with reference to a specific embodiment below.

[0025] 1. Overall overview of the plan.

[0026] The embodiment of the present invention provides a general middleware system built on ZNS SSD, which mainly includes: the middleware consists of a target component module (DM-Target), a creation component module (Init), an address mapping module (Mapping Table), a Zone management module (Zone Manager), a garbage collection module (Garbage Collection) and a data recovery module (Recovery). The target component module is responsible for intercepting and processing I / O (input / output) requests from the upper system, and adapting to the sequential write characteristics of ZNS SSD by cloning the request and adding metadata specific to ZNS SSD; the creation component module is responsible for the initialization and resource allocation of the middleware system; the address mapping module is responsible for maintaining the dynamic mapping relationship between the logical address of the block device and the logical address of the physical zone; the zone management module is responsible for monitoring and managing the zone status of ZNS SSD; the garbage collection module performs data migration and zone reset operations by detecting the proportion of invalid data; after a system failure, the data recovery module restores the system's operating status by reading persistent metadata and logs to ensure data consistency.

[0027] The above-mentioned general middleware system is inserted into the Device Mapper layer of the Linux (an operating system kernel) system in the form of a component. It intercepts the block device access I / O request sent by the upper system, maps the block device logical address into the logical address of the Zone in the ZNS SSD, and then sends it to the ZNS SSD, so as to support the upper system to use the ZNS SSD. Figure 2 As shown in the figure, an overview of the overall storage system architecture including ZNS SSD is provided. Application layer (Application) applications transfer application data ( Figure 2 The thick solid line in the figure) is sent to the virtual file system layer (VFS), and after being managed and processed by the virtual file system, the application data and file system metadata ( Figure 2 The middleware system provided by the embodiment of the present invention reads application data and virtual file system metadata through the device mapper driver layer interface in the device mapping layer, modifies the metadata of the middleware system after processing ( Figure 2Indicated by the thin dotted line in the middle), and sent to the ZNS SSD driver layer (ZNS SSD Driver). In addition, the middleware provided in the embodiment of the present invention is responsible for managing the Zone partitions in the ZNS SSD, and divides them into three types of partition groups, namely: a data partition group (Data Zone) for storing application data, a journal partition group (Journal Zone) for storing temporary metadata, and a compressed journal partition group (Compacted Journal Zone) for permanent metadata storage.

[0028] In addition, the universal middleware system manages the reading and writing of zones and garbage collection by maintaining metadata such as the status, capacity, used space, and valid data bitmap of the zones in the ZNS SSD, thereby ensuring the normal use of the ZNS SSD.

[0029] 2. Detailed introduction of the plan.

[0030] like Figure 3 As shown, the overall architecture of the universal middleware system provided by the embodiment of the present invention is shown, and each module in the system is introduced in detail below.

[0031] 1. Create component modules.

[0032] In an embodiment of the present invention, a component module is created to initialize the middleware system and its various functional modules. By loading a user configuration file, the operating parameters of each module in the general middleware system are set, system resources are initialized and a scheduling queue is created to enable normal startup and operation of each module. Among them, system resources refer to the metadata and mapping tables of each module in the middleware system, and queues refer to queues for caching I / O requests.

[0033] 2. Target component module.

[0034] In the embodiment of the present invention, the target component module is mainly responsible for intercepting and processing requests from the upper system (for example, I / O requests), and ensuring that the requests comply with the sequential write rules of ZNS by parsing and cloning the requests and adapting the characteristics of the ZNS SSD.

[0035] Specifically: When the upper system issues an I / O request (bio), the target component module will first intercept the I / O request and parse the block device logical address (LBA), read / write operation type, data buffer pointer and other information. By calling the Zone management module, the current state of the ZNS SSD is detected, and according to the content of the I / O request, the appropriate Zone is selected to respond, and the address mapping module is called to map the block device logical address of the cloned I / O request to the corresponding Zone logical address, ensuring that the request can be correctly read and written to the corresponding location in the ZNS SSD.

[0036] When the target component module completes processing the I / O request, it calls the registered callback function and returns the result to the upper system. The callback function not only returns the execution status of the I / O request, but also provides other necessary information, such as whether the data is written or read correctly, the logical address of the zone to which it is written, etc.

[0037] I / O requests include write requests and read requests. For write requests, after the write operation is completed, the callback function will notify the upper-layer system that the I / O request has been successfully written to the ZNS SSD, and return the specific zone logical address and offset information written. If an error occurs (such as zone write failure or device failure), the callback function will return an error code for the upper-layer system to perform corresponding error processing.

[0038] 3. Zone management module.

[0039] In the embodiment of the present invention, the Zone management module is responsible for managing and maintaining the Zone information of each partition in the ZNS SSD, dynamically managing the allocation, switching and recovery of the Zone, ensuring that the rules of sequential writing are followed, and ensuring that the system maintains efficient operation under high concurrency of multiple tasks; at the same time, it cooperates with the address mapping module to ensure the consistency of Zone writing and ensure that each Zone can work normally.

[0040] In an embodiment of the present invention, the zone management module is initialized through the following steps: (1) creating a partition according to a data type (such as a data zone, a log zone, a compressed log zone, etc.), and initializing the metadata of the partition. The metadata of each partition includes: a list of zones in each state (a list of free zones, a list of used zones, a list of full zones, a list of zones being garbage collected, and a list of zones waiting for garbage collection), and the number of zones; (2) allocating a corresponding number of zones to each partition group according to the number of zones, and initializing the metadata information of the zone, such as the type of the zone, write pointer, id (identification number), capacity, number of valid pages, and bitmap, etc.

[0041] In the embodiment of the present invention, in order to identify the status information of the Zone, the Zone management module sets several states of the Zone, such as Figure 4 As shown, it mainly includes: free state (Free), active state (Activated), full state (Full), garbage collection state (Garbage) and invalid state (Invalid).

[0042] (1) A zone that has been initialized and assigned to a partition but has not yet been written is marked as free and is in the free zone list of each partition, ready to be used.

[0043] (2) After an idle zone is used, it will be converted to an active state and added to the partition's used zone list, which means that more data will be written to this zone in the future. The active zone can be read or written by the upper-level system.

[0044] (3) The Zone Management Module will detect the status of the write zone every time a write is performed. When the zone is full, the Zone Management Module will convert it to a full state. In the full state, the zone will close the write interface and only accept read requests.

[0045] (4) With the continuous use of the hard disk, since the ZNS SSD only supports sequential writing and does not support in-place data updates, when it is full, if the data in the Zone is updated remotely, it will become invalid data. When the invalid data increases and reaches the garbage collection threshold, the garbage collection mechanism will be triggered, and the Zone will be transferred to the garbage collection state, and the reading and writing of the upper system will not be supported.

[0046] (5) When garbage collection is completed, the Zone is converted to an invalid state, waiting to be received by a partition; or it is directly received by a partition and converted to an idle state.

[0047] Figure 4 The left part of the diagram shows the conversion logic between the above states, and the right part shows the space examples in each state. In the box on the right, the blank part indicates that no data has been written, the horizontal line filled part indicates the written data (Data), and the vertical line indicates invalid data (Invalid Data). In addition, in the full state, the data (Data) does not distinguish between valid data and invalid data. In the garbage collection state, the data (Data) refers to valid data.

[0048] like Figure 5 As shown in the figure, the Zone management module is also responsible for managing Zone partitions. All Zone partitions are divided into three partition groups according to the type of data stored, namely the log partition group for storing temporary metadata and the compressed log partition group for permanently storing metadata. Generally speaking, the above two partition groups contain at most two Zone partitions respectively; the third partition group is the data partition group, which is used to store actual application data. Figure 5Partition N in represents the number of the last partition. The two partition groups storing metadata, log partition group and compressed log partition group, belong to the extra space overhead (Over Provision) required for system management data. The purpose of partition group management is to select the partition in the correct partition group according to the data type (application data or metadata) when the data is flushed; the two log partition groups in the partition group will be determined during system initialization so that the data recovery module can correctly read the metadata in the log partition group to restore the state of the ZNS SSD.

[0049] 4. Address mapping module.

[0050] In the embodiment of the present invention, the address mapping module converts the logical address of the block device into the logical address of the Zone in the ZNS SSD. In order to be compatible with the existing virtual block device interface, the embodiment of the present invention will be responsible for converting the traditional random write request into a sequential write, so that the virtual block device appears as a continuous block device that supports random writes, hiding the sequential write limitation of the ZNS SSD.

[0051] In the address mapping module, a mapping table is maintained for the entire ZNS SSD, which records the mapping relationship between the block device logical address and the Zone logical address in the ZNS SSD. When the middleware system receives a new read request, the mapping table will find the corresponding table entry based on the block device logical address and convert the block device logical address to the Zone logical address; when the middleware system receives a new write request, the mapping table will record and update the mapping relationship between the requested block device logical address and the Zone logical address. The mapping table is stored in the memory. After the system is restarted, it is necessary to read the reverse mapping table from the log area of ​​the ZNS SSD and then reconstruct the mapping table.

[0052] In this embodiment, the address mapping module adopts a dynamic zone allocation algorithm. When the upper system issues a write request, it first selects a most suitable zone from the active zone list for writing, and stores the corresponding address mapping information in the mapping table. If there is no suitable zone (that is, no active zone), some zones are released for reuse through the garbage collection mechanism, and then the zone for response is selected from them. When a zone is full, the next active zone will be selected to continue writing.

[0053] In the embodiment of the present invention, a corresponding algorithm may be selected according to actual needs to select the most suitable zone. Generally speaking, the most suitable zone selected by the algorithm means that the life cycles of the data in the zones are as similar as possible.

[0054] In the embodiment of the present invention, a mapping table merging mechanism is also introduced. When a Zone is full, the reverse mapping relationship of the Zone is integrated into a compressed data block aligned with a specified size (for example, 4KB) and flushed to the specified compression log area.

[0055] 5. Garbage collection module.

[0056] In the embodiment of the present invention, the garbage collection module is responsible for managing the recovery of invalid data and the zone reset operation in the ZNS SSD. Since the zones of the ZNS SSD can only be written sequentially, and once the zone is full, it must be reset before it can be used again, the garbage collection module is designed to improve the efficiency of storage space use and reduce the write amplification effect. The module implements functions such as real-time monitoring of the zone, detection of invalid data, data migration and zone reset.

[0057] This middleware adopts an active garbage collection strategy. The garbage collection module detects the proportion of invalid data in each zone by combining with the mapping table. When a block device logical address is deleted or overwritten (i.e., the zone data is updated off-site), the mapping table will mark it as invalid. The garbage collection module periodically (for example, every second) scans all zones and counts the proportion of valid data and invalid data in each zone; if the proportion of invalid data in the current zone reaches the threshold of garbage collection (for example, the proportion of invalid data exceeds 95%), the garbage collection mechanism is triggered; the valid data in the current zone is migrated to a new zone in an idle state. After the migration of valid data is completed, the current zone is restored to its initial state through the zone reset command (ResetZone) of the ZNS SSD. Once the reset is completed, it will become invalid, or the zone will be added to the list of idle zones, waiting for new write operations.

[0058] Preferably, in order to improve the efficiency of garbage collection, the garbage collection module adopts an optimization strategy. When selecting a recycling zone, the zone with the highest proportion of invalid data is given priority for garbage collection to maximize the efficiency of space release.

[0059] The garbage collection module in this embodiment significantly improves the space utilization and write performance of the ZNS SSD, reduces the write amplification effect, and effectively extends the service life of the device through an intelligent recycling trigger mechanism, an optimized data migration strategy, and a multi-level cache mechanism.

[0060] 6. Data recovery module.

[0061] In the embodiment of the present invention, the data recovery module is used to handle abnormal interruptions caused by system failures or abnormal conditions. The data recovery module ensures that after the system restarts, it can recover data consistency, restore the mapping table status, and reschedule unfinished I / O requests from the fault state. Through logging, transactional operations, and error detection mechanisms, the data recovery module can effectively ensure the data security and system reliability of the ZNS SSD.

[0062] When the system starts or restarts, the first step of the data recovery module is to restore the metadata of the ZNS SSD. This includes the mapping table (including the dynamic mapping relationship between the logical address of the block device maintained by the address mapping module and the logical address of the zone in the ZNS SSD), the zone usage status, the invalid data mark and the garbage collection information. In order to ensure the consistency of the metadata, when the system is running normally, the corresponding metadata will be flushed to the log area of ​​the ZNS SSD after each write request.

[0063] The data recovery module reads the saved metadata log (persistent log data), reloads the current state of the zone and the mapping table into memory, and ensures that the write offset of each zone is consistent with that before the system crash. If the metadata is found to be incomplete or conflicting during the recovery process, the system will automatically select the latest consistent version by comparing multiple versions of the log file.

[0064] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different modules to complete all or part of the functions described above.

[0065] Embodiment 2 The present invention also provides a method for implementing a general middleware system based on ZNS SSD, which is mainly used to implement the system provided by the aforementioned embodiment, and the method includes: setting a creation component module for initialization and resource allocation of the general middleware system; setting a target component module for intercepting and processing requests from the upper system, adding metadata of the ZNS SSD to adapt to the sequential writing characteristics of the ZNS SSD, calling the Zone management module, determining the Zone for response, and calling the address mapping module to map the block device logical address to the corresponding Zone logical address, and then sending it to the ZNS SSD for data operation; setting an address mapping module for maintaining the dynamic mapping relationship between the block device logical address and the Zone logical address in the ZNS SSD; setting a Zone management module for monitoring and managing the Zone status of each partition in the ZNS SSD, and selecting the Zone for response; setting a garbage collection module for detecting the proportion of invalid data in the Zone and performing data migration and Zone reset operations when the invalid data reaches the garbage collection threshold; setting a module for restoring the metadata information of the ZNS SSD by reading the persistent log data after a system failure; wherein the ZNS SSD is a space solid state drive, and Zone is a partition; based on the target component module, the creation component module, the address mapping module, the Zone management module, the garbage collection module and the data recovery module, a general middleware system based on ZNS SSD provided in the above embodiment is constructed.

[0066] Considering that the main technical details of each module involved in the method have been introduced in detail in the previous embodiments, they will not be repeated here.

[0067] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A general middleware system based on ZNS SSD, It is characterized in that include: Target component module, component creation module, address mapping module, zone management module, garbage collection module and data recovery module; among which: Create component modules for initialization and resource allocation of common middleware systems; The target component module is used to intercept and process requests from the upper system, add metadata of the ZNS SSD to adapt to the sequential write characteristics of the ZNS SSD, call the Zone management module, determine the Zone for response, and call the address mapping module to map the block device logical address to the corresponding Zone logical address, and then send it to the ZNS SSD for data operation; wherein the ZNS SSD is a space solid state drive and the Zone is a partition; The address mapping module is used to maintain the dynamic mapping relationship between the logical address of the block device and the logical address of the zone in the ZNS SSD; Zone management module, used to monitor and manage the zone status of each partition in ZNS SSD, and select the zone for response; The garbage collection module is used to detect the proportion of invalid data in the zone and perform data migration and zone reset operations when the invalid data reaches the garbage collection threshold; The data recovery module is used to restore the metadata information of the ZNS SSD by reading the persistent log data after a system failure.

2. According to claim 1, a general middleware system based on ZNS SSD, It is characterized in that The creation of a component module for initialization and resource allocation of a general middleware system includes: By loading the configuration file, the operating parameters of each module in the general middleware system are set, the system resources are initialized and the scheduling queue is created, so that each module can be started and run normally.

3. According to the general middleware system based on ZNS SSD in claim 1, It is characterized in that The zone states include: idle state, active state, full state, garbage collection state and invalid state; where: A Zone that has been initialized and assigned to a partition but has not been written to is marked as idle. After an idle Zone is used, it will be converted to an active state. After a Zone is fully written, it will be converted to a full state. After being in a full state, if the data in the Zone is updated remotely, it will become invalid data. When the amount of invalid data increases and reaches the threshold of garbage collection, the garbage collection mechanism will be triggered and the Zone will be converted to a garbage collection state. When garbage collection is completed, the Zone will be converted to an invalid state, waiting to be received by a partition, or directly received by a partition and converted to an idle state.

4. According to claim 3, a general middleware system based on ZNS SSD, It is characterized in that It also includes the initialization steps of the Zone management module: Create partitions according to data types and initialize the partition metadata. The metadata of each partition includes: a list of zones in each state and the number of zones; According to the number of zones, a corresponding number of zones are allocated to each partition group, and metadata information of the zones is initialized.

5. According to claim 4, a general middleware system based on ZNS SSD, It is characterized in that The partition groups include: a log partition group for storing temporary metadata, a compressed log partition group for permanently storing metadata, and a data partition group for storing application data; through partition group management, the partition in the correct partition group is selected according to the data type when the data is refreshed; the log partition group and the compressed log partition group in the partition group are determined when the middleware system is initialized, and are used for the data recovery module to read the corresponding data to restore the metadata information of the ZNS SSD.

6. A general middleware system based on ZNS SSD according to claim 1 or 3, It is characterized in that The Zone selected for response includes: A dynamic zone allocation algorithm is used to select a zone for response from active zones. If a zone cannot be selected, some zones are released through the garbage collection mechanism, and then a zone for response is selected from them.

7. According to claim 1, a general middleware system based on ZNS SSD, It is characterized in that The address mapping module is used to maintain the dynamic mapping relationship between the block device logical address and the zone logical address in the ZNS SSD, including: The address mapping module maintains a mapping table for the ZNS SSD, which records the mapping relationship between the block device logical address and the Zone logical address in the ZNS SSD; when the middleware system receives a new read request, it searches for the corresponding table entry according to the block device logical address and converts the block device logical address into the Zone logical address; when the middleware system receives a new write request, it records and updates the mapping relationship between the block device logical address and the Zone logical address; After the middleware system is restarted, the reverse mapping table is read from the log area of ​​the ZNS SSD, and then the mapping table is reconstructed.

8. According to claim 1, a general middleware system based on ZNS SSD, It is characterized in that The garbage collection module is used to detect the invalid data ratio in the zone and perform data migration and zone reset operations when the invalid data reaches the garbage collection threshold, including: The garbage collection module periodically scans all zones and counts the ratio of valid data to invalid data in each zone; if the ratio of invalid data in the current zone reaches the garbage collection threshold, the garbage collection mechanism is triggered; the valid data in the current zone is migrated to a new zone in an idle state. After the migration of valid data is completed, the current zone is restored to its initial state through a reset command.

9. According to claim 1, a general middleware system based on ZNS SSD, It is characterized in that The data recovery module is used to recover the metadata information of the ZNS SSD by reading the persistent log data after a system failure, including: The persistent log data includes: the dynamic mapping relationship between the block device logical address and the Zone logical address in the ZNS SSD maintained by the address mapping module, the Zone status of each partition in the ZNS SSD monitored and managed by the Zone management module, and the invalid data mark and garbage collection information of the Zone; The data recovery module reads the persistent log data and reloads the information in the persistent log data into the memory, so that the write offset of each zone remains consistent with that before the failure.

10. A method for implementing a general middleware system based on ZNS SSD. It is characterized in that include: Setting up a creation component module for initialization and resource allocation of a general middleware system; Set up to intercept and process requests from the upper system, add metadata of ZNS SSD to adapt to the sequential write characteristics of ZNS SSD, call the Zone management module, determine the Zone for response, and call the address mapping module to map the block device logical address to the corresponding Zone logical address, and then send it to the target component module of ZNS SSD for data operation; Set up an address mapping module for maintaining a dynamic mapping relationship between a block device logical address and a zone logical address in a ZNS SSD; set up a zone management module for monitoring and managing the zone status of each partition in the ZNS SSD and selecting a zone for response; Set up a garbage collection module that detects the proportion of invalid data in the zone and performs data migration and zone reset operations when the invalid data reaches the garbage collection threshold; It is configured to restore the metadata information of the ZNS SSD by reading the persistent log data after a system failure; wherein the ZNS SSD is a space solid state drive, and Zone is a partition; Based on the target component module, the creation component module, the address mapping module, the Zone management module, the garbage collection module and the data recovery module, a general middleware system based on the ZNS SSD as described in any one of claims 1 to 9 is constructed.

Citation Information

Patent Citations

  • Garbage collection method and device

    CN112306896A

  • Data writing method and device for partition namespace solid state disk and electronic equipment

    CN115951839A

  • ZNS-SSD storage system-oriented garbage collection method

    CN116301576A

  • L2P mapping table reconstruction method and solid state disk

    CN116467224A

  • Universal platform application support for plugin modules

    US20180341499A1

Cited By

  • Dynamic mapping strategy for improving space utilization rate of ZNS solid state disk

    CN119806416A

  • Internal object processing method and device, storage medium, electronic equipment and program product

    CN120045129A

  • Method, device, storage medium, electronic device and program product for processing internal objects

    CN120045129B