Storage system capable of reducing file fragments, read-write method and controller

Abstract

The invention discloses a storage system capable of reducing file fragments, a read-write method and a controller, and belongs to the technical field of data storage, the system comprises a host and a solid state disk, a file system and a space pre-distributor are created in the host, and a segment distributor is arranged in the solid state disk; the file system allocates a logic address space for the file according to segments; the segment distributor is used for maintaining a global translation table, the global translation table is used for recording a mapping relation from a segment ID to a mapping page address, and the mapping page is a flash memory page used for recording a logic page address to a physical page address in the segment; the segment distributor is further used for calculating idle segment IDs in a monotone increasing mode and issuing the idle segment IDs to the ring buffer area when the ring buffer area has residual space; and the space pre-distributor is used for taking out the idle segment ID of the head of the annular buffer area and distributing the idle segment ID to the file when the space needs to be distributed to the file. The generation of file fragments and free space fragments in the file system can be reduced, and the overall performance and stability of the system are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and more specifically, relates to a storage system, a reading and writing method and a controller for reducing file fragmentation. Background Art

[0002] Solid State Disk (SSD) has the characteristics of fast read and write, light weight, low energy consumption and small size, and is increasingly widely used in storage solutions. SSD uses an internal flash translation layer (FTL) to manage the mapping relationship between logical block address space (LBA space) and physical flash space (PBA space). SSD usually uses an array structure to implement FTL. Although this structure is simple to implement and easy to manage, it will cause the size of the mapping table to be proportional to the logical address space, so it cannot support larger logical address spaces.

[0003] In a storage system based on SSDs, the space allocation of the file system in the host usually adopts an on-demand allocation strategy, which means that when the file data is written, it will be allocated to the free logical address space (LBA space) provided by the SSD. This strategy can reduce space waste, but when multiple threads write multiple files concurrently at the same time, the data of different files may be stored interleaved with each other, making the data blocks of the same file discontinuously distributed in the logical address space, thereby generating file fragmentation. The existence of file fragmentation will seriously affect subsequent sequential read and update operations, reducing the overall performance and stability of the system.

[0004] In addition, due to the existence of file fragmentation, when a file is deleted, discrete free blocks will be left in the original space, namely free space fragments. These free space fragments will cause new files to be unable to find sufficiently large continuous space when they are created, thereby aggravating the generation of file fragmentation. Summary of the invention

[0005] In view of the defects of the prior art and the need for improvement, the present invention provides a storage system, a read-write method and a controller for reducing file fragmentation, the purpose of which is to reduce the generation of file fragments and free space fragments in the file system and improve the overall performance and stability of the storage system based on the solid state drive.

[0006] To achieve the above object, according to one aspect of the present invention, a storage system for reducing file fragmentation is provided, comprising a host and a solid state drive;

[0007] A file system is created in the host; the file system allocates logical address space for files in units of fixed-size segments; a segment includes multiple logical pages;

[0008] The host is also provided with a space pre-allocator, which is used to maintain a ring buffer in the host's memory; the ring buffer is used to store segment IDs;

[0009] The firmware of the solid state drive is provided with a segment allocator; the segment allocator is used to maintain a global conversion table; the global conversion table is used to record the mapping relationship from the segment ID to the mapping page address; the mapping page is a flash memory page in the solid state drive used to record the logical page address within the segment to the physical page address;

[0010] The segment allocator is further used to calculate the free segment ID in a monotonically increasing manner and determine the mapping page to which it is mapped when there is remaining space in the ring buffer, record the mapping relationship in the mapping table, and publish the calculated free segment ID to the tail of the ring buffer;

[0011] The space pre-allocator is also used to take out the free segment ID in the head of the ring buffer and allocate it to the file when the file system needs to allocate space for the file.

[0012] Furthermore, the hash value of the segment ID is the address of the mapping page to which it is mapped.

[0013] Furthermore, the segment allocator calculates the free segment ID in a monotonically increasing manner and determines the mapped page to which it is mapped, including:

[0014] S1: Get the segment ID at the end of the ring buffer, add 1 to its value, and use it as the candidate segment ID;

[0015] S2: Calculate the hash value of the candidate segment ID to determine the address of the mapping page to which the candidate segment ID is mapped;

[0016] S3: If the remaining space of the mapping page mapped to the candidate segment ID is lower than the average remaining space of the mapping page, the value of the candidate segment ID is increased by 1 and used as the new candidate segment ID, and then transferred to S2; otherwise, the current candidate segment ID is used as the calculation result, and its hash value is used as the mapped mapping page address.

[0017] Furthermore, the segment allocator is also used to access the global translation table to obtain the mapping page address to which the segment is mapped when the file system writes the logical page in the segment to the solid state drive, and record the mapping relationship from the logical page address to the physical page address in the mapping page.

[0018] According to one aspect of the present invention, a storage system reading and writing method for reducing file fragmentation is provided, including a file writing operation; the file writing operation includes:

[0019] W1: Write the file F to be written according to the logical page size w Divide into multiple file blocks;

[0020] W2: In the logical address space of the file system, it is the address of file F. w Assign a segment ID and write the file blocks to the corresponding segment in sequence until all the file blocks are written or the assigned segment is full;

[0021] W3: If all file blocks are written, transfer to W4; otherwise, after writing the file blocks in the allocated segment to the SSD, transfer to W2;

[0022] W4: File write operation completed.

[0023] Furthermore, the reading and writing method provided by the present invention further includes: a file reading operation; the file reading operation includes:

[0024] R1: Get the segment S where the file content to be read is located R ;

[0025] R2: Get segment S from the mapping table R Mapped page address:

[0026] R3: Access segment S according to the mapped page address R The mapped page to obtain segment S R The address of the physical page to which each logical page is mapped;

[0027] R3: Access segment S R The physical pages to which each logical page is mapped are used to obtain the content to be read, and the file read operation is completed.

[0028] According to another aspect of the present invention, there is provided a read / write controller of the storage system for reducing file fragmentation, comprising:

[0029] A computer-readable storage medium for storing a computer program;

[0030] And a processor, used to read the computer program stored in the computer-readable storage medium and execute the above-mentioned reading and writing method provided by the present invention.

[0031] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0032] (1) When the present invention allocates a logical address space for a file, it allocates a segment each time, and each segment contains multiple logical pages, thereby realizing space reservation, ensuring that even if the file needs to be expanded in subsequent use, its data layout can still maintain continuity within the segment, thereby effectively avoiding the generation of file fragments; the ID of the allocable segment is generated in a monotonically increasing manner, thereby providing an extended logical address space for the file system, and this extended logical address space design allows the logical address space to not be exhausted during the entire life cycle of the solid-state hard disk; at the same time, the present invention adopts a global conversion table to record the mapping relationship, and realizes a two-level mapping structure, the first level records the mapping relationship from the logical segment address to the second level mapping page address, and the second level records the mapping relationship from each logical page in the segment to the physical page, effectively reducing the storage space required for the mapping table, avoiding excessive storage overhead of the mapping structure due to the expansion of the logical address space, and improving the access efficiency of the solid-state hard disk. In general, the present invention pre-allocates continuous free space by the host-side file system, and provides an extended LBA space on the device side, combined with the optimization of the mapping structure, effectively reducing the generation of file fragments in the file system, and improving the performance and stability of the storage system based on the solid-state hard disk.

[0033] (2) The global conversion table maintained by the present invention is a hash-based mapping structure that can achieve high performance and high efficiency for sequential access.

[0034] (3) In the present invention, when determining the free segment ID, the segment allocator on the device side will comprehensively consider the remaining space of the mapping page to which it is mapped, and when the remaining space is lower than the average remaining space of the mapping page, the segment ID will be skipped and the next segment ID will be considered. This ensures the balance between the mapping pages and improves the access efficiency of the mapping relationship in the mapping page. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a storage system for reducing file fragmentation provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the logical layout of a file system provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a file read operation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0039] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0040] In order to solve the technical problem that the traditional file system allocates the free logical address space provided by the solid-state hard disk on demand, which will form file fragments and reduce the overall performance and stability of the system, the present invention provides a storage system, a read-write method and a controller for reducing file fragments. The overall idea is to provide an extended logical address space for the file system by modifying the internal mapping mechanism of the solid-state hard disk, and the logical address space will not be exhausted during the entire life cycle of the solid-state hard disk; based on the extended logical address space, the file system allocates logical address space to the file in units of larger logical segments, realizes space reservation, and ensures the continuity of the file data layout, thereby reducing the generation of file fragments and improving the overall performance and stability of the system.

[0041] The following are examples.

[0042] Embodiment 1:

[0043] A storage system that reduces file fragmentation, such as Figure 1 As shown, it includes a host and a solid state drive; a file system is created in the host, a space pre-allocator is also set in the host, and a segment allocator is set in the firmware of the solid state drive.

[0044] like Figure 2As shown, in this embodiment, the logical partition of the file system is divided into three parts: a file system metadata area, a file metadata area and a data area; wherein the file system metadata area is used to record the information of all solid-state hard disks in the system; the file metadata area is used to store the basic information of each file; and the data area is used to store file data. Different from the traditional file system based on solid-state hard disk (SSD) (such as EXT4, F2FS, etc.), in this embodiment, in the data area of ​​the file system, space is allocated for files in segments of fixed size, and different files are allocated different segments. When the file exceeds the size of a segment, a new segment is allocated again. Correspondingly, the LBA space provided by the solid-state hard disk is equally divided into several segments of the same size. Normally, in a solid-state hard disk, the size of a physical page is 4KB; the size of the segment should be set to ensure that the read operation within the segment can be regarded as a sequential read. Optionally, in this embodiment, the size of the segment is 1MB.

[0045] In this embodiment, a logical segment is allocated to a file each time, which ensures the continuity of the file data layout and effectively reduces the generation of file fragments.

[0046] like Figure 1 As shown, in this embodiment, the space pre-allocator is used to maintain a ring buffer in the host's memory; the ring buffer is used to store segment IDs. The size of the ring buffer can be set accordingly according to the load of the file system.

[0047] like Figure 1 As shown, in this embodiment, the segment allocator is used to maintain the global conversion table; the global conversion table is used to record the mapping relationship from the segment ID to the mapping page address; the mapping page is a flash memory page in the solid state drive used to record the logical page address within the segment to the physical page address.

[0048] In this embodiment, the global conversion table maintained by the segment allocator implements a two-level mapping structure. The first level records the mapping relationship from the logical segment address to the second-level mapping page address, and the second level records the mapping relationship from each logical page in the segment to the physical page. Compared with the traditional array-based mapping structure, this embodiment effectively reduces the storage space required for the mapping table and improves the access efficiency of the solid-state drive.

[0049] In order to achieve high performance and high efficiency for sequential access, as a preferred implementation, in this embodiment, the hash value of the segment ID is the address of the mapping page to which it is mapped.

[0050] In this embodiment, the segment allocator is also used to calculate the free segment ID in a monotonically increasing manner and determine the mapped mapping page when there is remaining space in the ring buffer, record the mapping relationship in the mapping table, and publish the calculated free segment ID to the end of the ring buffer.

[0051] In this embodiment, the space pre-allocator is also used to take out the free segment ID in the head of the ring buffer and allocate it to the file when the file system needs to allocate space for the file.

[0052] In this implementation, the segment allocator generates segment IDs in a monotonically increasing manner and publishes them to the ring buffer, ensuring that the segment IDs stored in the ring buffer are all unused new segments; therefore, when the space pre-allocator allocates logical segments to files, it also allocates new segments in a monotonically increasing manner, avoiding repeated allocation of used logical segments and avoiding the overhead caused by recording space allocation conditions.

[0053] In the process of issuing and retrieving the above logical segment ID, the host-side space pre-allocator is equivalent to a consumer, which is only responsible for reading free segment IDs from the ring buffer, and the device-side segment allocator is equivalent to a producer, which is only responsible for adding new segment IDs to the ring buffer. The host-side space pre-allocator and the device-side segment allocator cooperate with each other to separate the logical address space of the SSD from the physical NAND space, so that the host-side file system can pre-allocate unlimited continuous space for files without considering the actual use of the physical space, thereby improving storage efficiency and access speed.

[0054] This embodiment provides an extended logical address space for the file system by modifying the internal mapping mechanism of the solid-state drive and allocating logical segments to files in a monotonically increasing manner. In practical applications, the size of the logical space provided by the solid-state drive to the file system through the segment allocator is determined by the number of bits of the logical address. If the logical address is recorded as an integer, then in a system with a 64-bit integer, a maximum of 8ZByte of logical address space can be supported. According to the actual use of the solid-state drive, the logical address space will not be exhausted during the entire life cycle of the solid-state drive, which makes it possible for the file system to reserve space for files in segments. At the same time, this embodiment uses a global conversion table to record the mapping relationship, which effectively reduces the storage space required for the mapping table and avoids excessive storage overhead of the mapping structure due to the expansion of the logical address space.

[0055] In order to further improve the access performance of the system, in this embodiment, the device side segment allocator will comprehensively consider the remaining space of the mapping page when generating an unused new segment ID. Specifically, the segment allocator calculates the free segment ID in a monotonically increasing manner and determines the mapping page to which it is mapped, including:

[0056] S1: Get the segment ID at the end of the ring buffer, add 1 to its value, and use it as the candidate segment ID;

[0057] S2: Calculate the hash value of the candidate segment ID to determine the address of the mapping page to which the candidate segment ID is mapped;

[0058] S3: If the remaining space of the mapping page mapped to the candidate segment ID is lower than the average remaining space of the mapping page, the value of the candidate segment ID is increased by 1 and used as the new candidate segment ID, and then transferred to S2; otherwise, the current candidate segment ID is used as the calculation result, and its hash value is used as the mapped mapping page address.

[0059] by Figure 1 For example, the segment ID calculation process is as follows:

[0060] Get the segment ID at the end of the ring buffer, add 1 to its value, and get the candidate segment ID (for example, 0x1FF);

[0061] The hash value of the candidate segment ID is calculated to be 1023, and the remaining space of the corresponding mapping page is obtained to be 2KB, which is lower than the average remaining space of the mapping page. The segment ID is skipped and the next candidate segment ID (for example, 0x200) is calculated.

[0062] The hash value of the candidate segment ID is calculated to be 1021, and the remaining space of the corresponding mapping page is obtained to be 3.2KB, which is not less than the average remaining space of the mapping page. The candidate segment ID is used as the segment ID calculation result, and the mapping relationship between the segment ID and the mapping page it maps is recorded in the global conversion table, and the segment ID is published to the tail of the ring buffer.

[0063] The device-side segment allocator calculates and publishes the ID of the free segment in the above manner, thereby ensuring the balance between the mapping pages and improving the access efficiency of the mapping relationship in the mapping page.

[0064] In this embodiment, the segment allocator is also used to access the global translation table to obtain the mapping page address to which the segment is mapped when the file system writes the logical page in the segment to the solid state drive, and record the mapping relationship from the logical page address to the physical page address in the mapping page.

[0065] In general, this embodiment expands the logical address space of the file system by modifying the internal mapping mechanism of the SSD, thereby avoiding the exhaustion of the logical address space throughout the life cycle of the SSD. This method allows the file system to pre-allocate a fixed-size free space window for each file, ensuring the continuity of the data layout when the file is subsequently expanded, and effectively avoiding the generation of file fragments. This not only improves the storage efficiency of the SSD, but also significantly improves the stability and performance of the file system, and has significant practical value for application scenarios that require processing large amounts of data and high-concurrency write operations.

[0066] Embodiment 2:

[0067] A reading and writing method is applied to the storage system for reducing file fragmentation provided in the above-mentioned embodiment 1.

[0068] The read-write method provided in this embodiment includes a file write operation; the file write operation includes:

[0069] W1: Write the file F to be written according to the logical page size w Divide into multiple file blocks;

[0070] W2: In the logical address space of the file system, it is the address of file F. w Assign a segment ID and write the file blocks to the corresponding segment in sequence until all the file blocks are written or the assigned segment is full;

[0071] W3: If all file blocks are written, transfer to W4; otherwise, after writing the file blocks in the allocated segment to the SSD, transfer to W2;

[0072] W4: File write operation completed.

[0073] The read and write method provided in this embodiment also includes: file read operation; Figure 3 As shown, the file read operation includes:

[0074] R1: Get the segment S where the file content to be read is located R ;

[0075] R2: Get segment S from the mapping table R Mapped page address:

[0076] R3: Access segment S according to the mapped page address R The mapped page to obtain segment S R The address of the physical page to which each logical page is mapped;

[0077] R3: Access segment S R The physical pages to which each logical page is mapped are used to obtain the content to be read, and the file read operation is completed.

[0078] Embodiment 3:

[0079] A read-write controller is applied to the storage system for reducing file fragmentation provided in the above-mentioned embodiment 1.

[0080] This embodiment includes:

[0081] A computer-readable storage medium for storing a computer program;

[0082] And a processor, used to read the computer program stored in the computer-readable storage medium and execute the reading and writing method provided in the above embodiment 2.

[0083] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A storage system for reducing file fragmentation, comprising a host and a solid state drive; characterized in that: A file system is created in the host; the file system allocates a logical address space for a file in units of segments of a fixed size; a segment includes a plurality of logical pages; The host is also provided with a space pre-allocator, and the space pre-allocator is used to maintain a ring buffer in the memory of the host; the ring buffer is used to store the segment ID; A segment allocator is provided in the firmware of the solid state drive; The segment allocator is used to maintain a global conversion table; the global conversion table is used to record the mapping relationship from segment ID to mapping page address; the mapping page is a flash memory page in the solid state drive used to record the logical page address within the segment to the physical page address; The segment allocator is further used to calculate the free segment ID in a monotonically increasing manner and determine the mapped mapping page when there is remaining space in the ring buffer, record the mapping relationship in the mapping table, and publish the calculated free segment ID to the tail of the ring buffer; The space pre-allocator is also used to take out the free segment ID in the head of the ring buffer and allocate it to the file when the file system needs to allocate space for the file.

2. The storage system for reducing file fragmentation according to claim 1, characterized in that: The hash value of the segment ID is the address of the mapping page to which it is mapped.

3. The storage system for reducing file fragmentation according to claim 2, characterized in that: The segment allocator calculates the free segment ID in a monotonically increasing manner and determines the mapped page to which it is mapped, including: S1: Get the segment ID at the end of the ring buffer, add 1 to its value, and use it as a candidate segment ID; S2: Calculate the hash value of the candidate segment ID to determine the address of the mapping page to which the candidate segment ID is mapped; S3: If the remaining space of the mapping page mapped to the candidate segment ID is lower than the average remaining space of the mapping page, the value of the candidate segment ID is increased by 1 and used as the new candidate segment ID, and then transferred to S2; otherwise, the current candidate segment ID is used as the calculation result, and its hash value is used as the mapped mapping page address.

4. The storage system for reducing file fragmentation according to any one of claims 1 to 3, characterized in that: The segment allocator is also used to access the global conversion table to obtain the mapping page address to which the segment is mapped when the file system writes the logical page in the segment to the solid state drive, and record the mapping relationship from the logical page address to the physical page address in the mapping page.

5. The storage system reading and writing method for reducing file fragmentation according to any one of claims 1 to 4, characterized in that: Including file writing operation; the file writing operation includes: W1: Write the file F to be written according to the logical page size w Divide into multiple file blocks; W2: in the logical address space of the file system, it is file F w Assign a segment ID and write the file blocks to the corresponding segment in sequence until all the file blocks are written or the assigned segment is full; W3: if all file blocks are written, transfer to W4; otherwise, after writing the file blocks in the allocated segment to the SSD, transfer to W2; W4: The file writing operation is completed.

6. The reading and writing method according to claim 5, characterized in that: Also includes: File read operation; The file reading operation includes: R1: Get the segment S where the file content to be read is located R ; R2: Get segment S from the mapping table R Mapped page address: R3: Access segment S according to the mapped page address R The mapped page to obtain segment S R The address of the physical page to which each logical page is mapped; R3: Access segment S R The physical pages to which each logical page is mapped are obtained to obtain the content to be read, and the file read operation is completed.

7. The read / write controller of a storage system for reducing file fragmentation according to any one of claims 1 to 3, characterized in that: include: A computer-readable storage medium for storing a computer program; And a processor, used to read the computer program stored in the computer-readable storage medium and execute the reading and writing method described in claim 5 or 6.

Images