A data storage system and method

Abstract

The application discloses a data storage system and method, the system comprises: an application module, a heterogeneous file module and a ZNS storage module, the ZNS storage module is provided with a plurality of isolated areas; the heterogeneous file module is used for receiving user instruction information sent by the application module, analyzing the user instruction information to obtain write request information, and sending storage space configuration information to the ZNS storage module according to the write request information; the ZNS storage module is used for configuring the data partition type of each isolated area according to the storage space configuration information. The application can divide a plurality of isolated areas in the storage module, support a user to issue instructions to configure the isolated areas correspondingly to adapt to the needs of different users, further separate sequential write data and random write data in the isolated areas, thereby enhancing the data read-write performance, improving the space utilization of data storage and realizing good system full-load continuous performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular to a data storage system and method. BACKGROUND

[0002] In the prior art, the data storage scheme for the ZNS (zoned namespace) device is generally F2FS or BTRFS file system. Among them, F2FS is a file system optimized and designed for flash characteristics, adopts a log-based write mode, meets the sequential write constraint of ZNS, adjusts the mapping rule to align the logical granularity and the physical granularity; and the BTRFS based on write-time copy introduces the concept of "log super block" on the basis of the original architecture, solves the in-place update problem of the super block, and therefore supports the ZNS device.

[0003] However, the data isolation storage strategy cannot be implemented in the multi-tenant scenario in the prior art, and redundant metadata overhead is generated. SUMMARY

[0004] In order to solve the above technical problems, the embodiments of the present application provide a data storage system and method, which can improve the read-write performance, improve the storage space utilization rate and have better continuous performance when full.

[0005] In order to achieve the above purpose, the embodiments of the present application provide a data storage system, which comprises an application module, a heterogeneous file module and a ZNS storage module, the ZNS storage module is provided with a plurality of isolated areas;

[0006] The heterogeneous file module is used for receiving user instruction information sent by the application module, analyzing the user instruction information to obtain write request information, and sending storage space configuration information to the ZNS storage module according to the write request information.

[0007] The ZNS storage module is used for configuring the data partition type of each isolated area according to the storage space configuration information.

[0008] Further, the write request information includes sequential write request information, random write request information, or mixed request information, and the storage space configuration information includes sequential configuration information, regular configuration information, or mixed configuration information; then, the heterogeneous file module is configured to receive user instruction information sent by the application module, parse the user instruction information to obtain write request information, and send storage space configuration information to the ZNS storage module according to the write request information, specifically including: when the write request information is the sequential write request information, sending the sequential configuration information to the ZNS storage module; when the write request information is the random write request information, sending the regular configuration information to the ZNS storage module; and when the write request information is the mixed request information, sending the mixed configuration information to the ZNS storage module.

[0009] Further, the isolation region includes a first isolation region and a second isolation region; then, the ZNS storage module is configured to configure the data partition type of each isolation region according to the storage space configuration information, specifically including: configuring the data partition type of each isolation region as a sequential partition type according to the sequential configuration information; or configuring the data partition type of each isolation region as a regular partition type according to the regular configuration information; or configuring the data partition type of the first isolation region as a sequential partition type and configuring the data partition type of the second isolation region as a regular partition type according to the mixed configuration information.

[0010] Further, the heterogeneous file module includes a super partition unit, a directory partition unit, and a data partition unit; the super partition unit is configured to store device information, isolation region information, directory address information, and consistency checkpoint information; and the directory partition unit is configured to store directory node information and directory data.

[0011] Further, the super partition unit is provided with a partition manager; the partition manager is configured to store partition group information, partition descriptor information, partition linked list information, and partition address table information.

[0012] Further, the data partition unit includes a regular data partition and a sequential data partition; wherein the regular data partition is provided with a shielding underlying mapping mechanism, and the sequential data partition is provided with a sequential index mapping mechanism.

[0013] Further, the heterogeneous file module is provided with a file heterogeneity management mechanism.

[0014] Further, the file heterogeneity management mechanism includes a sequential write file heterogeneity management mechanism and a random write file heterogeneity management mechanism.

[0015] Further, the data storage system is provided with a garbage data recycling mechanism.

[0016] The embodiment of the present application also provides a data storage method, applied to any of the data storage systems, comprising the following steps:

[0017] The heterogeneous file module receives the user instruction information sent by the application module, parses the user instruction information to obtain write request information, and sends storage space configuration information to the ZNS storage module according to the write request information.

[0018] The ZNS storage module configures the data partition type of each isolated region according to the storage space configuration information.

[0019] In summary, the present application has the following beneficial effects:

[0020] In the storage module, a plurality of isolated regions are divided, and the isolated regions are configured by the user to adapt to the needs of different users, and the sequential write data and the random write data are further separated in the isolated region, and the garbage data recycling mechanism is configured, so that the data read-write performance in the multi-user scene is enhanced, the space utilization of data storage is improved, and good system full-load continuous performance is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a structural schematic diagram of one embodiment of the data storage system provided by the present application;

[0022] Figure 2 is a flow schematic diagram of one embodiment of the data storage method provided by the present application;

[0023] Figure 3 is a data interaction schematic diagram of one embodiment of the data storage system provided by the present application;

[0024] Figure 4 is a sequential write file heterogeneous management mechanism schematic diagram of one embodiment of the data storage system provided by the present application;

[0025] Figure 5 is a random write file heterogeneous management mechanism schematic diagram of one embodiment of the data storage system provided by the present application;

[0026] Figure 6 is a random write file heterogeneous management mechanism schematic diagram of another embodiment of the data storage system provided by the present application;

[0027] Figure 7 is a random write file heterogeneous management mechanism schematic diagram of another embodiment of the data storage system provided by the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0029] Referring to Figure 1 is a flowchart of an embodiment of the data storage system provided by the present application. The system 102 includes an application module 201, a heterogeneous file module 202 and a ZNS storage module 203. The ZNS storage module 203 sets a plurality of isolated areas.

[0030] The heterogeneous file module 202 is configured to receive user instruction information sent by the application module 201, analyze the user instruction information to obtain write request information, and send storage space configuration information to the ZNS storage module 203 according to the write request information.

[0031] Preferably, the heterogeneous file module 202 includes a super partition unit, a directory partition unit and a data partition unit. The super partition unit is configured to store device information, isolated area information, directory address information and consistency checkpoint information. The directory partition unit is configured to store directory node information and directory data.

[0032] Preferably, the super partition unit is provided with a partition manager. The partition manager is configured to store partition group information, partition descriptor information, partition link list information and partition address table information.

[0033] Preferably, the data partition unit includes a regular data partition and a sequential data partition. The regular data partition is provided with a shielding bottom layer mapping mechanism, and the sequential data partition is provided with a sequential index mapping mechanism.

[0034] For example, the overall architecture of the heterogeneous file module is as follows: the zone is the smallest space management unit. The zones of the heterogeneous file module are strictly aligned with the "zone" of the ZNS device. According to the content stored in the zone, the zone of the heterogeneous file module is defined as three types:

[0035] 1) Super zone. Store general device information, partition manager, isolated area information, directory address and consistency checkpoint. The super zone contains two super block backups. Once a crash occurs, the system can recover from the other super block to the state before the update in time.

[0036] 2) directory zone. The directory related nodes and data are stored in directory zone, which is separated from files. On one hand, it can speed up the mounting process, without the need to scan the whole disk to build the file tree. On the other hand, it can speed up the directory search and update. The directory is frequently updated content, which is updated in memory, combined and modified, periodically updated, and the performance impact of recursive update is reduced.

[0037] 3) data zone. The data zone has two types: conventional (CNV) type and sequential (SEQ) type. The conventional data zone hides the write order of the underlying medium through the mapping mechanism, so it supports random write and sequential write like a conventional block device, while the sequential data zone simplifies the mapping rule to balance performance and overhead, and only provides sequential indexing, so it only supports sequential write.

[0038] For example, each isolated area has a corresponding zone manager, which contains two different processing flows inside, respectively processing the requests sent to the SEQ zone and the CNV zone. When the request is distributed to the isolated area, the zone manager will select a flow according to the path to schedule and interact with the file interface and the device interface. Referring to Figure 3 , the zone manager mainly includes the following contents:

[0039] Definition 1. Zone group information. The zone group records the scheduling information of the same type of data zone, such as the sequence number range, the current access point, and the zone list index.

[0040] Definition 2. Zone descripter. The zone descripter is the unique identifier of the zone, which records the state of the zone, the sequential write point, and the number of files and valid blocks contained. The zone state has four types: empty, full, explicitly opened, and implicitly opened, among which explicitly opened represents that the zone can be normally written, implicitly opened represents that the zone is performing recycling writing and needs to wait for the end of the cleaning process, and the zone can be normally written after it is converted to the explicitly opened state.

[0041] Definition 3. Zone list. There are two kinds of zone list: allocated list and free list. Allocated list manages full or open zone. Free list manages empty zone. For SEQ zone, list adopts round robin allocation, allocation and reclaim happen at the head and tail of list. For CNV zone, it needs to determine the zone to be allocated and reclaimed based on selection strategy. The difference of list management of different zone groups is determined by index mechanism and file characteristics: SEQ zone is based on sequential index, the modification frequency of file is low, and round robin can improve efficiency and save overhead. CNV zone adopts a more flexible mapping mechanism, the modification frequency of the file stored is high, and the state of the zone and the number of valid blocks are always in dynamic change, so a more complex selection sorting strategy is needed.

[0042] Definition 4. Zone address table (ZAT). The essence of ZAT is an array, the length of the array is equal to the number of CNV zones, the zone number is the array offset, and the memory address of the zone node (Znode) corresponding to the zone is stored in the array. Znode is an intermediate node of ZAT and allocated list, which records the reclaim order of the zone and maintains the allocation order of the zone. Since the random search efficiency of list is low, ZAT and Znode can be used to access the zone list more flexibly and efficiently, and improve the garbage collection efficiency.

[0043] When a file is about to be stored in the "SEQ" directory or the "CNV" directory, the zone manager will first check the allocated list. If the zone being used at the tail of the list is not full, the space requested by the file will be allocated in sequence, and the starting address of the space will be returned through the file interface. If the remaining space of the current zone is insufficient, the reclaim program will be started, and after the reclaim is completed, the allocation will be performed again.

[0044] Preferably, the write request information includes sequential write request information, random write request information, or mixed request information, and the storage space configuration information includes sequential configuration information, regular configuration information, or mixed configuration information; the heterogeneous file module 202 is configured to receive user instruction information sent by the application module 201, parse the user instruction information to obtain write request information, and send storage space configuration information to the ZNS storage module 203 according to the write request information, specifically including: when the write request information is the sequential write request information, sending the sequential configuration information to the ZNS storage module 203; when the write request information is the random write request information, sending the regular configuration information to the ZNS storage module 203; and when the write request information is the mixed request information, sending the mixed configuration information to the ZNS storage module 203.

[0045] For example, the heterogeneous file module divides an isolated area for each tenant, and the tenant can configure the type and number of data partitions in the isolated area:

[0046] 1) If the tenant only issues sequential write requests, all data partitions in the current isolated area are used as SEQ (sequential) type, and only sequential write files are stored. For a log update scenario, the tenant can obtain greater performance gain and lower system overhead.

[0047] 2) If the tenant only issues random write requests, all data partitions in the current isolated area are used as CNV (conventional) type, and random write files can be received. For a frequent file modification scenario, this setting can provide the tenant with a larger logical space for coverage and simplify the modification of the application.

[0048] 3) If the tenant issues mixed sequential and random write requests, the tenant can configure the number of SEQ and CNV partitions by itself, allowing the tenant to achieve greater data separation.

[0049] Preferably, the heterogeneous file module 202 is provided with a file heterogeneity management mechanism.

[0050] Preferably, the file heterogeneity management mechanism includes a sequential write file heterogeneity management mechanism and a random write file heterogeneity management mechanism.

[0051] For example, in order to reduce the migration cost of the upper-layer application, the heterogeneous file module retains the inode and dentry-based file heterogeneity management mechanism, and hides the write order of the underlying device from the upper-layer application through file mapping. The file structure is composed of inode blocks and data blocks, and the inode includes file metadata, inline data, and extent index. The extent records the starting point and offset of the continuous data block. However, as an end-to-end file system, the heterogeneous file module has greater freedom in software and hardware cooperation. When designing the mapping mechanism, the host-side memory and CPU resources should be minimized, for example, for applications that support sequential writing, providing a too complex mapping mechanism will lead to performance surplus and resource waste. For applications that do not support sequential writing, a robust mapping mechanism must be provided to cope with different application architectures for file modification. Therefore, on the basis of data isolation, the heterogeneous file module designs a heterogeneous management mechanism for sequential write files and random write files: (1) Sequential write files are stored in SEQ partitions, which support sequential writing and append writing of files. Based on this, an extent-based sequential index structure is designed, as shown in Figure 4Each time the write is added, an extent structure pointing to the data segment is added at the end of the index. (2) The random write file is stored in the CNV partition, and the dynamic change of the extent link table is used to handle the random update of the file. The position of the covered write generates an invalid data block, and the heterogeneous file module does not mark the invalid data block, but only updates the valid data by modifying the link table record. For details, see Figure 5 If the coverage area involves multiple extents, the completely invalid extent will be directly removed from the index link table, and the related memory structure is released. The start point and the end point of the extent are discarded by modifying the start point and the length of the extent. If the file modification range is within an extent, see Figure 6 , the extent is divided into discontinuous data segments, and the remaining valid data segments are re-established.

[0052] The ZNS storage module 203 is configured to configure the data partition type of each isolated region according to the storage space configuration information.

[0053] Preferably, the isolated region includes a first isolated region and a second isolated region; then, the ZNS storage module 203 is configured to configure the data partition type of each isolated region according to the storage space configuration information, specifically including: according to the order configuration information, the data partition type of each isolated region is configured as a sequential partition type; or, according to the regular configuration information, the data partition type of each isolated region is configured as a regular partition type; or, according to the mixed configuration information, the data partition type of the first isolated region is configured as a sequential partition type, and the data partition type of the second isolated region is configured as a regular partition type.

[0054] Preferably, the data storage system 102 is provided with a garbage data recycling mechanism.

[0055] Exemplarily, when garbage collection or index space is insufficient, extent merging occurs, see Figure 7 The partition to be recycled indirectly realizes defragmentation, space integration, and re-releases the index structure during data migration. Each update of the file will modify the metadata. For frequently modified files, an inode cache based on the LRU strategy is set to accumulate metadata updates through the cache, thereby improving the file update efficiency. In addition, the cache will periodically write back the inode block of the hot data as a verification node for data recovery and garbage collection.

[0056] See Figure 2 is a flowchart of an embodiment of the data storage method provided by the application, which includes steps S1 to S2, as follows:

[0057] S1, the isomerization file module receives the user instruction information sent by the application module, parses the user instruction information to obtain write request information, and sends storage space configuration information to the ZNS storage module according to the write request information;

[0058] S2, the ZNS storage module configures the data partition type of each isolated region according to the storage space configuration information.

[0059] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary hardware platform, and of course, it can also be implemented entirely by hardware. Based on such understanding, all or part of the technical solutions of the present application which contribute to the background art can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment or some part of the embodiment of the present application.

[0060] In summary, the present application has the following beneficial effects:

[0061] In the storage module, a plurality of isolated regions are divided, and the user is supported to issue instructions to configure the isolated regions accordingly to adapt to the needs of different users. In addition, the sequential write data and the random write data are further separated in the isolated region, and a garbage data recycling mechanism is configured, thereby enhancing the data read-write performance in a multi-user scenario, improving the space utilization of data storage, and achieving good system full-load continuous performance.

[0062] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A data storage system, characterized by, The application module, the heterogeneous file module and the ZNS storage module are included, and the ZNS storage module is provided with a plurality of isolated areas; The heterogeneous file module is used for receiving user instruction information sent by the application module, analyzing the user instruction information to obtain write request information, and sending storage space configuration information to the ZNS storage module according to the write request information; The ZNS storage module is used for configuring the data partition type of each isolated area according to the storage space configuration information; The heterogeneous file module includes a super partition unit, and the super partition unit is provided with a partition manager; the partition manager is used for storing partition group information, partition descriptor information, partition chain table information and partition address table information; The partition group information is used for recording scheduling information of the same type of data partition; Each isolated area corresponds to a partition manager, and the heterogeneous file module divides an isolated area for each tenant; The write request information includes sequential write request information, random write request information or mixed request information, and the storage space configuration information includes sequential configuration information, regular configuration information or mixed configuration information; Then, the heterogeneous file module is used for receiving user instruction information sent by the application module, analyzing the user instruction information to obtain write request information, and sending storage space configuration information to the ZNS storage module according to the write request information, specifically including: When the write request information is the sequential write request information, the sequential configuration information is sent to the ZNS storage module; When the write request information is the random write request information, the regular configuration information is sent to the ZNS storage module; When the write request information is the mixed request information, the mixed configuration information is sent to the ZNS storage module.

2. The data storage system of claim 1, wherein, The isolated area includes a first isolated area and a second isolated area; Then, the ZNS storage module is used for configuring the data partition type of each isolated area according to the storage space configuration information, specifically including: According to the sequential configuration information, the data partition type of each isolated area is configured as a sequential partition type; Or, According to the regular configuration information, the data partition type of each isolated area is configured as a regular partition type; Or, According to the mixed configuration information, the data partition type of the first isolated area is configured as a sequential partition type, and the data partition type of the second isolated area is configured as a regular partition type.

3. The data storage system of claim 1, wherein, The heterogeneous file module further includes a directory partition unit and a data partition unit; The super partition unit is used for storing device information, isolated area information, directory address information and consistency checkpoint information; The directory partition unit is used for storing directory node information and directory data.

4. The data storage system of claim 3, wherein, The data partition unit includes a regular data partition and a sequential data partition; wherein the regular data partition is provided with a shielding bottom layer mapping mechanism, and the sequential data partition is provided with a sequential index mapping mechanism.

5. The data storage system of claim 1, wherein, The heterogeneous file module is provided with a file heterogeneous management mechanism.

6. The data storage system of claim 5, wherein, The file heterogeneous management mechanism includes a sequential write file heterogeneous management mechanism and a random write file heterogeneous management mechanism.

7. The data storage system of claim 1, wherein, The data storage system is provided with a garbage data recycling mechanism.

8. A data storage method applied to the data storage system according to any one of claims 1 to 7, characterized in that, The method comprises the following steps: The heterogeneous file module receives the user instruction information sent by the application module, parses the user instruction information to obtain write request information, and sends storage space configuration information to the ZNS storage module according to the write request information. The ZNS storage module configures the data partition types of each isolated region according to the storage space configuration information.

Images