High-performance bare disk management method

Abstract

The present invention provides a high-performance bare disk management method, including: dividing the disk space into a user data area and a metadata area, dividing the user data area into multiple slices, and the metadata information of the slices is located in the metadata area; the user module is responsible for maintaining the mapping relationship between data and data block management units and data block management units and slice IDs, and sending I / O requests to slice control modules; the slice control module is used to perform disk control, and the slice control module is a multi-layer structure, and each layer performs corresponding control operations; wherein the disk control includes disk space management, data consistency control, and I / O operations, and the data consistency control is performed through a clock mechanism. The present invention has the following beneficial effects: improving management efficiency, reducing metadata operations, reducing kernel interactions, improving software performance, and avoiding single-disk performance bottlenecks caused by CPU reasons.

Description

Technical Field

[0001] The invention belongs to the field of communication technology, and in particular relates to a high-performance bare disk management method. Background Art

[0002] Most of the existing distributed storage systems use common local file systems (such as EXT4, XFS, etc.) to manage disks at the bottom layer to reduce development and maintenance costs. However, due to the universality of local file systems, there are inevitably overly complex designs and unnecessary I / O overheads. For example, metadata operations of most file systems are not necessary for distributed storage systems; although some distributed storage systems use self-developed bare disk management systems, most of them are limited in applicability and have various constraints.

[0003] The general local file system is mostly designed in kernel mode, and has numerous metadata operations, which severely limits performance. Other self-developed bare disk management systems on the market are mostly in interrupt mode and cannot fully unleash the performance of nvme disks. Summary of the invention

[0004] In view of this, the present invention aims to propose a high-performance bare disk management method, in order to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] A high-performance bare disk management method, the method comprising:

[0007] Divide the disk space into a user data area and a metadata area, divide the user data area into multiple shards, and the metadata information of the shards is located in the metadata area;

[0008] Maintain the mapping relationship between data and data block management unit and between data block management unit and shard ID through the user module, and send the I / O request to the shard control module;

[0009] Disk control is performed through a shard control module, which has a multi-layer structure, and each layer performs corresponding control operations; wherein, the disk control includes disk space management, data consistency control, and I / O operations, and the data consistency control is performed through a clock mechanism.

[0010] Furthermore, the metadata area is divided into a disk information tool part and a shard array part:

[0011] The disk information tool part is used to store disk-level verification information and basic disk management information;

[0012] The slice array part is a structure array, and each structure in the structure array corresponds to metadata information of a slice.

[0013] Furthermore, the mapping relationship between the user data area and the metadata area is managed through a bitmap and a hash table; each pixel in the bitmap corresponds to a data block management unit; the key of the hash table is the data block management unit ID, and the value is the offset and clock information within the slice.

[0014] Furthermore, the process of managing through the bitmap and hash table includes:

[0015] Allocation: Find the free shards through the bitmap and mark them as used, create a data block management unit entity and record its offset in the hash table;

[0016] Release: Search the data block management unit entity corresponding to the data block management unit ID through the hash table, update the bitmap according to the offset in the data block management unit entity and mark it as unused, and remove the information of the data block management unit entity from the hash table.

[0017] Furthermore, the multi-layer structure of the shard control module specifically includes:

[0018] A slicing layer, which is used to manage the allocation and release of disk space, and the mapping relationship between data block management units and disk space;

[0019] A logic layer, which is used to process data consistency logic and convert upper-layer I / O requests into offsets on actual disks;

[0020] The execution layer is responsible for interacting with the disk and performing actual I / O operations.

[0021] Furthermore, each shard corresponds to an entity, and the entity falls into a unique core according to a hash rule, and each core has multiple entities; the hash rule is: calculate the sum of the hard disk ID and the shard serial number, and take the modulus of the sum and the number of cores.

[0022] Furthermore, the process of performing data consistency control through the clock mechanism includes:

[0023] Each data block management unit entity is created when applying for space and initializes its clock information to 0; write operations increase the clock value, while read operations do not change the clock value.

[0024] During each write operation, the user module adds 1 to the current clock value and passes this value along with the I / O request to the shard control module; during a read operation, the user module passes the currently maintained clock value along with the I / O request to the shard control module.

[0025] Furthermore, the process of performing data consistency control through the clock mechanism also includes, for write operations:

[0026] The shard control module compares the clock in the I / O request with the clock in the data block management unit entity; if the clock in the I / O is greater than the clock in the data block management unit entity, the I / O is submitted for write operation, and the clock in the I / O is updated to the data block management unit entity, and a write success message is returned; otherwise, a write failure message is returned;

[0027] After receiving the write success message, the user module updates the current clock to the clock in the I / O.

[0028] Furthermore, the process of performing data consistency control through the clock mechanism also includes, for read operations:

[0029] The shard control module checks whether the clock in the I / O is equal to the clock in the data block management unit entity; if they are equal, the I / O is valid and the read operation is performed; otherwise, the I / O is expired and a failure message is returned.

[0030] Furthermore, the software stack of the method is a combination of polling and coroutine, and uses a full user-mode software design.

[0031] Compared with the prior art, the high-performance bare disk management method of the present invention has the following beneficial effects:

[0032] The extremely simple data management method uses bitmap+hash table to improve management efficiency, reduce metadata operations, and provide higher iops to the upper layer; it adopts full user mode to reduce kernel interaction and improve software performance; the polling and lock-free mechanism is more friendly to high-performance nvme devices; the single-disk multi-core design can better exert the performance of the nvme disk and avoid the performance bottleneck of the single disk due to CPU reasons. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0034] Figure 1 This is a schematic diagram of the structural division of the user module and the disk management module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0036] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0037] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0038] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0039] Some of the professional terms involved in the embodiments of the present invention include:

[0040] Fass - distributed all-flash block storage system.

[0041] BACTL - Local disk management system for Fass;

[0042] Chunk: data block management unit of fass, usually 16M;

[0043] Chkid: Each chunk corresponds to a unique id;

[0044] Diskid: Each disk has a unique ID.

[0045] Sid: sharding serial number in the disk;

[0046] Dsid: Each disk is divided into multiple shardings, and each sharding has a unique dsid in the system. It consists of diskid+sid;

[0047] Clock: version number of data write operation;

[0048] Bitmap: Bitmap, used to mark the usage of each bit;

[0049] Hashtable: Hash table, used to maintain a mapping relationship from key to value;

[0050] Polling: Computer terminology, a CPU operation mechanism;

[0051] Core: Computer professional term, a core of a CPU;

[0052] Coroutine: A computer term for a program component that can be understood as a lightweight user-mode thread.

[0053] High-performance bare disk management method, including:

[0054] The disk space is divided into a user data area and a metadata area, and the user data area is divided into multiple shards. The metadata information of the shards is located in the metadata area.

[0055] The metadata area is divided into a disk information tool part and a shard array part:

[0056] The disk information tool part is used to store disk-level verification information and basic disk management information; the shard array part is a structure array, and each structure in the structure array corresponds to the metadata information of a shard.

[0057] The mapping relationship between the user data area and the metadata area is managed through a bitmap and a hash table.

[0058] Each pixel in the bitmap corresponds to a data block management unit; the key of the hash table is the data block management unit ID, and the value is the offset within the slice and clock information.

[0059] In some embodiments, a specific scheme for partitioning the disk space is as follows:

[0060] The disk space of BACTL is generally divided into the metadata area and the user data area. The metadata area records the metadata information of the disk, which is divided into two parts: the first part is Diskinfo, which stores the disk-level verification information and basic disk management information; the second part is sharding_array, which is a structure array. The number of arrays corresponds to the number of shardings divided in the Data area. Each structure represents the metadata information of a sharding.

[0061] The user data area stores the user's real data. According to the metadata information, the data area is divided into multiple shardings. Each sharding area corresponds to a metadata information in sharding_array.

[0062] The data mapping relationship of BACTL is managed in sharding units. The data mapping relationship of each sharding is managed in the form of Bitmap+Hashtable. Bitmap is used to record the usage of disk space, and each bit represents the usage of a chunk, i.e., 16M space. Hashtable is used to record the offset of the upper-level data block corresponding to the sharding. The key is chkid, and the value stores the offset and clock information in the sharding.

[0063] Diskinfo includes:

[0064] md_begin, md_end: records the start and end offsets of the metadata area.

[0065] data_begin,data_end: records the start and end offset positions of the data area;

[0066] Sharding_count: the number of shards in the data area of ​​the disk;

[0067] Sinfo_begin, Sinfo_end: represents the start and end offset positions of sharding_array in the MetaData storage area;

[0068] The structure of a sharding in Sharding_array is sharding_info, including: sharding_begin, sharding_end corresponding to the start and end positions of the sharding management data area.

[0069] The process of management through bitmaps and hash tables includes:

[0070] Allocation: Find the free shards through the bitmap and mark them as used, create a data block management unit entity and record its offset in the hash table;

[0071] Release: Search the data block management unit entity corresponding to the data block management unit ID through the hash table, update the bitmap according to the offset in the data block management unit entity and mark it as unused, and remove the information of the data block management unit entity from the hash table.

[0072] In some embodiments, a specific solution for management through bitmap and hash table is: the Sharding entity mainly maintains Bitmap and Hashtable information, wherein Bitmap is used to maintain the usage of sharding space, and Hashtable is used to maintain the correspondence between chkid and chunk entity, wherein the chunk entity mainly records the offset and clock information in the sharding corresponding to the chunk.

[0073] Allocation logic: When Sharding receives a disk space allocation request, it finds a free location based on the Bitmap information in the sharding entity and marks the location as 1, indicating that it is used; creates the corresponding chunk entity, and records the offset information allocated by the Bitmap in the chunk entity, and adds the chunk entity to the Hashtable of the sharding entity.

[0074] Release logic: When Sharding receives a disk space release request, it finds the chunk entity information corresponding to chkid from the Hashtable of the sharding entity; gets the offset from the chunk entity information. According to the offset operation, the Bitmap is set to 0, indicating that the space is unused; and the chunk entity information is deleted from the Hashtable of the sharding entity.

[0075] The mapping relationship between data and data block management unit and between data block management unit and shard ID is maintained through the user module, and the I / O request is sent to the shard control module.

[0076] Disk control is performed through the shard control module. The shard control module has a multi-layer structure, and each layer performs corresponding control operations. Among them, disk control includes disk space management, data consistency control, and I / O operations. Data consistency control is performed through the clock mechanism.

[0077] The multi-layer structure of the shard control module specifically includes:

[0078] Sharding layer: The sharding layer is used to manage the allocation and release of disk space, as well as the mapping relationship between data block management units and disk space;

[0079] Logical layer: The logical layer is used to process data consistency logic and convert upper-layer I / O requests into offsets on actual disks.

[0080] Execution layer: The execution layer is responsible for interacting with the disk and performing actual I / O operations.

[0081] like Figure 1 As shown, in some embodiments, a specific solution for constructing and dividing the user module and the slicing control module is as follows:

[0082] Rangeectl of fass is used as the user layer. The mapping relationship between data and chunk and the relationship between chunk and dsid are maintained in Rangeectl. Based on this relationship, the io request is sent to BACTL.

[0083] BACTL is mainly divided into sharding layer, logical layer, and driver layer:

[0084] The Sharding layer is mainly responsible for the allocation and release of disk space and the management of the mapping relationship between chkid and disk space;

[0085] The Logical layer is mainly responsible for processing data consistency logic and converting upper-layer IO into the actual offset position on the disk;

[0086] The Driver layer is mainly responsible for interacting with the disk and performing actual IO operations on the disk.

[0087] Taking the write operation as an example, an io process based on the above embodiment is given:

[0088] After the Sharding layer receives the io, it finds the corresponding chunk entity in the hash_table of the sharding entity according to the chkid information, and sends the io and chunk entity to the logical layer;

[0089] After receiving the request, the Logical layer determines the clock information carried in the io and the clock information stored in the chunk entity. If the clock information carried in the io is greater than the clock information in the chunk entity, it is a valid write and is allowed to pass. After calculating the offset, the io is sent to the driver layer. If the clock information carried in the io is less than or equal to the clock information in the chunk entity, it is an invalid write and returns failure.

[0090] Offset calculation method:

[0091] offset=io->offset+chunk->offset+sharding_begin;

[0092] The converted IO is sent to the driver layer. If the driver layer succeeds, it returns successfully. When it returns successfully to the logical layer, the clock of the write operation is updated to the chunk entity. After the driver layer receives the real offset of the disk, it calls the spdk interface to perform specific IO operations on the disk in user mode.

[0093] The correspondence between disks and cores is: each shard corresponds to an entity, and the entity falls into a unique core according to the hash rule. There are multiple entities in each core; the hash rule is: calculate the sum of the hard disk ID and the shard serial number, and take the sum modulo the number of cores.

[0094] In some embodiments, the correspondence between the disk and the core specifically includes:

[0095] There is a many-to-many correspondence between disks and cores; each core can have multiple sharding entities;

[0096] A disk can be divided into multiple shardings, each sharding corresponds to a sharding entity, and the sharding entity falls on a unique core according to the hash rule. The hash rule is: (diskid+sid)%core_count;

[0097] When adding a disk to a fass cluster, specify the sharding count, and persist the sharding count to the disk Diskinfo metadata information, which cannot be changed. For example, if bactl is configured with 4 cores, responsible for 2 disks, and each disk is divided into 8 shardings, the 8 sharding entities of each disk will be evenly distributed on the 4 cores. There will be 4 sharding entities on each core.

[0098] The process of performing data consistency control through the clock mechanism includes:

[0099] Each data block management unit entity is created when applying for space and initializes its clock information to 0; write operations increase the clock value, while read operations do not change the clock value.

[0100] During each write operation, the user module adds 1 to the current clock value and passes this value along with the I / O request to the shard control module; during a read operation, the user module passes the currently maintained clock value along with the I / O request to the shard control module.

[0101] For write operations: the shard control module compares the clock in the I / O request with the clock in the data block management unit entity; if the clock in the I / O is greater than the clock in the data block management unit entity, the I / O is submitted for write operation, and the clock in the I / O is updated to the data block management unit entity, and a write success message is returned; otherwise, a write failure message is returned; after receiving the write success message, the user module updates the current clock to the clock in the I / O.

[0102] For read operations: the shard control module checks whether the clock in the I / O is equal to the clock in the data block management unit entity; if they are equal, the I / O is valid and the read operation is performed; otherwise, the I / O is expired and a failure message is returned.

[0103] In some embodiments, the specific execution process of data consistency control is:

[0104] The data order consistency guarantee scheme is implemented by maintaining the clock of the chunk entity:

[0105] When a chunk applies for space, it creates a chunk entity. The chunk entity records the offset and clock information. The initial clock is 0, and it increases by 1 for each write operation, but remains unchanged for read operations.

[0106] Rangectl also maintains the clock information of each chunk: the write operation will add 1 to the currently maintained clock and put it in io to pass it to BACTL; the read operation uses the currently maintained clock and puts it in io to pass it to BACTL;

[0107] BACTL receives io and judges the clock at the logical layer:

[0108] If it is a write operation, the clock in io and the clock in chunk entity are judged; if the clock in io is greater than the clock in chunk entity, io is valid, and io is submitted to the driver layer. After the write is successful, the clock in io is updated to the chunk entity, and a success message is returned; if the clock in io is less than or equal to the clock in chunk entity, io is expired, and a failure message is returned; after Rangectl receives the write operation success message, it updates and maintains the clock information to the clock submitted in io.

[0109] If it is a read operation, determine whether the clock in the io is equal to the clock in the chunk entity; if they are equal, the io is valid and the io is submitted to the driver layer; if they are not equal, it is an expired io and a failure message is returned.

[0110] The software stack of the solution described in the embodiment of the present invention adopts the polling+coroutine method, which reduces context switching. From the reception of IO by the foreground until the driver layer calls SPDK, full-path and full-memory operations are performed without locks or context switching, thereby improving software performance. After the driver layer calls SPDK, the coroutine yield method is adopted. During the process of polling and waiting for SPDK to complete the processing of IO, other IO operations can be continued, thereby giving full play to the performance of the CPU.

[0111] In addition, a full user-mode software design is used. All BACTL programs run in user mode to avoid software performance bottlenecks caused by user-mode / kernel-mode switching. The driver layer uses spdk to drive user-mode disks, avoiding switching when accessing kernel-mode disks.

[0112] Compared with the traditional local file system, the solution of the present invention adopts a very simple data management method, adopts the bitmap+hash table method for management, improves management efficiency, reduces metadata operations, and provides higher iops for the upper layer; adopts a full user mode to reduce kernel interaction and improve software performance; the Polling and lock-free mechanism is more friendly to high-performance nvme devices; the single-disk multi-core design can better exert the performance of the nvme disk and avoid the single-disk performance bottleneck caused by CPU reasons.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A high-performance bare disk management method, characterized in that: The method comprises: Divide the disk space into a user data area and a metadata area, divide the user data area into multiple shards, and the metadata information of the shards is located in the metadata area; Maintain the mapping relationship between data and data block management unit and between data block management unit and shard ID through the user module, and send the I / O request to the shard control module; The disk control is performed by a shard control module, the shard control module is a multi-layer structure, and each layer performs corresponding control operations; wherein the disk control includes disk space management, data consistency control, and I / O operations, and the data consistency control is performed by a clock mechanism; The mapping relationship between the user data area and the metadata area is managed through a bitmap and a hash table; each pixel in the bitmap corresponds to a data block management unit; the key of the hash table is the data block management unit ID, and the value is the offset and clock information within the slice; The process of performing data consistency control through the clock mechanism includes: Each data block management unit entity is created when applying for space, and its clock information is initialized to 0; write operations increase the clock value, while read operations do not change the clock value; the user module adds 1 to the current clock value during each write operation and passes this value to the shard control module along with the I / O request; the user module passes the currently maintained clock value to the shard control module along with the I / O request during the read operation; The process of executing data consistency control through the clock mechanism also includes, for write operations: the shard control module compares the clock in the I / O request with the clock in the data block management unit entity; if the clock in the I / O is greater than the clock in the data block management unit entity, the I / O is submitted for write operation, and the clock in the I / O is updated to the data block management unit entity, and a write success message is returned; otherwise, a write failure message is returned; after receiving the write success message, the user module updates the current clock to the clock in the I / O; The process of performing data consistency control through the clock mechanism also includes, for read operations: the shard control module checks whether the clock in the I / O is equal to the clock in the data block management unit entity; if they are equal, the I / O is valid and the read operation is performed; otherwise, the I / O is expired and a failure message is returned.

2. The high-performance bare disk management method according to claim 1, characterized in that: The metadata area is divided into a disk information tool part and a slice array part: The disk information tool part is used to store disk-level verification information and basic disk management information; The slice array part is a structure array, and each structure in the structure array corresponds to metadata information of a slice.

3. The high-performance bare disk management method according to claim 1, characterized in that: The process of managing through bitmap and hash table includes: Allocation: Find the free shards through the bitmap and mark them as used, create a data block management unit entity and record its offset in the hash table; Release: Search the data block management unit entity corresponding to the data block management unit ID through the hash table, update the bitmap according to the offset in the data block management unit entity and mark it as unused, and remove the information of the data block management unit entity from the hash table.

4. The high-performance bare disk management method according to claim 1, characterized in that: The multi-layer structure of the shard control module specifically includes: A slicing layer, which is used to manage the allocation and release of disk space, and the mapping relationship between data block management units and disk space; A logic layer, which is used to process data consistency logic and convert upper-layer I / O requests into offsets on actual disks; The execution layer is responsible for interacting with the disk and performing real I / O operations; After the driver layer receives the real offset of the disk, it calls the spdk interface to perform specific io operations on the disk in user mode.

5. The high-performance bare disk management method according to claim 1, characterized in that: Each shard corresponds to an entity, and the entity falls into a unique core according to a hash rule, and each core has multiple entities; the hash rule is: calculate the sum of the hard disk ID and the shard serial number, and take the modulus of the sum and the number of cores.

6. The high-performance bare disk management method according to claim 1, characterized in that: The software stack of the method is a combination of polling and coroutine, and uses a full user-mode software design.

Images