A storage system, read / write method, and controller for reducing file fragmentation.

Abstract

This invention discloses a storage system, read / write method, and controller for reducing file fragmentation, belonging to the field of data storage technology. The system includes a host and a solid-state drive (SSD). The host has a file system and a space pre-allocator, while the SSD has a segment allocator. The file system allocates logical address space to files according to segments. The segment allocator maintains a global translation table, which records the mapping relationship between segment IDs and mapped page addresses. The mapped pages are flash pages used to record the logical page addresses within a segment to the physical page addresses. The segment allocator also calculates free segment IDs in a monotonically increasing manner and publishes them to the circular buffer when there is remaining space. The space pre-allocator retrieves the free segment ID from the head of the circular buffer and allocates it to the file when space needs to be allocated. This invention can reduce the generation of file fragmentation and free space fragmentation in the file system, improving the overall performance and stability of the system.

Description

Technical Field

[0001] This invention belongs to the field of data storage technology, and more specifically, relates to a storage system, read / write method, and controller for reducing file fragmentation. Background Technology

[0002] Solid-state drives (SSDs) are increasingly widely used in storage solutions due to their fast read / write speeds, lightweight design, low power consumption, and small size. SSDs use an internal flash translation layer (FTL) to manage the mapping between the logical block address space (LBA) and the physical block address space (PBA). SSDs typically implement the FTL using an array structure. While this structure is simple to implement and easy to manage, it results in the mapping table size being proportional to the logical address space, thus limiting its ability to support larger logical address spaces.

[0003] In solid-state drive (SSD) based storage systems, the file system in the host typically uses an on-demand allocation strategy. This means that file data is allocated to the free logical address space (LBA space) provided by the SSD when it is written. This strategy can reduce space waste, but when multiple threads concurrently write to multiple files, data from different files may be interleaved, causing data blocks of the same file to be non-contiguously distributed in the logical address space, resulting in file fragmentation. The existence of file fragmentation can severely affect subsequent sequential read and update operations, reducing the overall performance and stability of the system.

[0004] Furthermore, due to the existence of file fragmentation, when a file is deleted, the original space will be left with discrete free blocks, namely free space fragments. These free space fragments will cause new files to be unable to find a sufficiently large contiguous space when they are created, thus exacerbating the generation of file fragmentation. Summary of the Invention

[0005] In view of the shortcomings of existing technologies and the need for improvement, this invention provides a storage system, read / write method and controller for reducing file fragmentation. Its purpose is to reduce the generation of file fragments and free space fragments in the file system and improve the overall performance and stability of solid-state drive-based storage systems.

[0006] To achieve the above objectives, according to one aspect of the present invention, a storage system for reducing file fragmentation is provided, comprising a host and a solid-state drive;

[0007] A file system is created in the host; the file system allocates logical address space for files in units of fixed-size segments; a segment consists of multiple logical pages;

[0008] The host also has a space pre-allocator, which is used to maintain a circular buffer in the host's memory; the circular buffer is used to store segment IDs.

[0009] The firmware of a solid-state drive (SSD) contains a segment allocator; the segment allocator is used to maintain a global translation table; the global translation table is used to record the mapping relationship between segment IDs and mapped page addresses; the mapped page is a flash page in the SSD used to record the logical page address to the physical page address within a segment.

[0010] The segment allocator is also used to calculate the free segment ID in a monotonically increasing manner and determine the mapped page when there is remaining space in the circular buffer, record the mapping relationship in the mapping table, and publish the calculated free segment ID to the tail of the circular buffer.

[0011] The space pre-allocator is also used to extract the free segment ID from the head of the circular buffer and allocate it to the file when the file system needs to allocate space for the file.

[0012] Furthermore, the hash value of the segment ID is the address of the mapped page to which it is mapped.

[0013] Furthermore, the segment allocator calculates the free segment ID in a monotonically increasing manner and determines the mapped page, including:

[0014] S1: Obtain the segment ID at the end of the circular buffer, increment its value by 1, and use it as a candidate segment ID;

[0015] S2: Calculate the hash value of the candidate segment ID to determine the address of the mapped page to which the candidate segment ID is mapped;

[0016] S3: If the remaining space of the mapped page to which the candidate segment ID is mapped is lower than the average remaining space of the mapped pages, then increment the value of the candidate segment ID by 1 and use it as the new candidate segment ID, and proceed to S2; otherwise, use the current candidate segment ID as the calculation result and its hash value as the address of the mapped page.

[0017] Furthermore, the segment allocator is also used to access the global translation table to obtain the address of the mapped page to which the segment is mapped when the file system writes the logical pages in the segment to the solid-state drive, and to record the mapping relationship between the logical page address and the physical page address in the mapped page.

[0018] According to one aspect of the present invention, a read / write method for the above-described storage system for reducing file fragmentation is provided, including a file write operation; the file write operation includes:

[0019] W1: Write the file F to be written according to the logical page size. w Divide into multiple file blocks;

[0020] W2: In the logical address space of the file system, it is file F w Allocate a segment ID and write the file blocks sequentially to the corresponding segment until all file blocks are written or the allocated segment is full;

[0021] W3: If the entire file block is written, proceed to W4; otherwise, write the file blocks in the allocated segment to the solid-state drive and then proceed to W2.

[0022] W4: File write operation completed.

[0023] Furthermore, the read / write method provided by this invention further includes: a file read operation; the file read operation includes:

[0024] R1: Get the segment S containing the content of the file to be read. R ;

[0025] R2: Retrieve segment S from the mapping table R The mapped page address:

[0026] R3: Access segment S based on the mapped page address R The mapped page to obtain segment S R The addresses of the physical pages that each logical page in the table maps to;

[0027] R3: Access segment S R The file read operation ends when the logical pages in the file are mapped to the physical pages to retrieve the content to be read.

[0028] According to another aspect of the present invention, a read / write controller for the above-described storage system for reducing file fragmentation is provided, comprising:

[0029] A computer-readable storage medium for storing computer programs;

[0030] And a processor, used to read a computer program stored in a computer-readable storage medium and execute the read / write method provided by the present invention.

[0031] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0032] (1) When allocating logical address space for files, this invention allocates one segment at a time, with each segment containing multiple logical pages. This achieves space reservation, ensuring that even if the file needs to be expanded in subsequent use, its data layout can still maintain the continuity within the segment, thus effectively avoiding file fragmentation. The IDs of allocable segments are generated in a monotonically increasing manner, thus providing an extended logical address space for the file system. This extended logical address space design allows the logical address space to remain unused throughout the entire lifecycle of the solid-state drive. Simultaneously, this invention uses a global translation table to record mapping relationships, implementing a two-level mapping structure. The first level records the mapping relationship from logical segment addresses to the second level mapping page addresses, and the second level records the mapping relationship from each logical page within a segment to a physical page. This effectively reduces the storage space required for the mapping table, avoids excessive storage overhead due to logical address space expansion, and improves the access efficiency of the solid-state drive. In summary, this invention, through pre-allocating contiguous free space on the host-side file system and providing extended LBA space on the device side, combined with optimized mapping structures, effectively reduces file fragmentation in the file system and improves the performance and stability of solid-state drive-based storage systems.

[0033] (2) The global conversion table maintained by this invention is a hash-based mapping structure that can achieve high performance and high efficiency for sequential access.

[0034] (3) In this invention, when determining an idle segment ID, the segment allocator on the device side will comprehensively consider the remaining space of the mapped page. If the remaining space is lower than the average remaining space of the mapped page, the segment ID will be skipped and the next segment ID will be considered. This ensures the balance between the mapped pages and can improve the access efficiency of the mapping relationship in the mapped page. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a storage system for reducing file fragmentation provided in an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the logical layout of the file system provided in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of a file reading operation provided in an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0039] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0040] To address the technical problem that traditional file systems allocate free logical address space provided by solid-state drives (SSDs) on demand, leading to file fragmentation and reduced overall system performance and stability, this invention provides a storage system, read / write method, and controller to reduce file fragmentation. The overall approach involves modifying the internal mapping mechanism of the SSD to provide the file system with an extended logical address space that will not be exhausted throughout the SSD's lifetime. Based on this extended logical address space, the file system allocates logical address space to files in larger logical segments, reserving space and ensuring the continuity of file data layout, thereby reducing file fragmentation and improving overall system performance and stability.

[0041] The following is an example.

[0042] Example 1:

[0043] A storage system that reduces file fragmentation, such as Figure 1 As shown, it includes a host and a solid-state drive; the host has a file system created in it and a space pre-allocator is also set up in it, while the solid-state drive's firmware has a segment allocator set up in it.

[0044] like Figure 2As shown, in this embodiment, the logical partition of the file system is divided into three parts: the file system metadata area, the file metadata area, and the data area. The file system metadata area records information about all solid-state drives (SSDs) in the system; the file metadata area stores basic information about each file; and the data area stores file data. Unlike traditional SSD-based file systems (such as EXT4 and F2FS), in this embodiment, space is allocated to files in the file system's data area using fixed-size segments, with different segments allocated to different files. When a file exceeds the size of a segment, a new segment is allocated. Correspondingly, the LBA space provided by the SSD is equally divided into several segments of the same size. Typically, the physical page size in an SSD is 4KB; the segment size setting should ensure that read operations within a segment can be considered sequential reads. Optionally, in this embodiment, the segment size is 1MB.

[0045] In this embodiment, a logical segment is allocated to the file each time, which ensures the continuity of the file data layout and effectively reduces the generation of file fragments.

[0046] like Figure 1 As shown, in this embodiment, the space pre-allocator maintains a circular buffer in the host's memory; the circular buffer stores segment IDs. The size of the circular buffer can be set according to the file system load.

[0047] like Figure 1 As shown, in this embodiment, the segment allocator is used to maintain a global translation table; the global translation table is used to record the mapping relationship from segment ID to mapped page address; the mapped page is a flash page in the solid-state drive used to record the logical page address to the physical page address within a segment.

[0048] In this embodiment, the global translation table maintained by the segment allocator implements a two-level mapping structure. The first level records the mapping relationship from logical segment addresses to page addresses in the second level, and the second level records the mapping relationship from each logical page in the segment to a physical page. Compared with the traditional array-based mapping structure, this embodiment effectively reduces the storage space required by the mapping table and improves the access efficiency of the solid-state drive.

[0049] In order to achieve high performance and efficiency for sequential access, as a preferred implementation, in this embodiment, the hash value of the segment ID is the address of the mapped page to which it is mapped.

[0050] In this embodiment, the segment allocator is also used to calculate the free segment ID in a monotonically increasing manner and determine the mapped page when there is remaining space in the circular buffer, record the mapping relationship in the mapping table, and publish the calculated free segment ID to the tail of the circular buffer.

[0051] In this embodiment, the space pre-allocator is also used to extract the free segment ID from the head of the circular buffer and allocate it to the file when the file system needs to allocate space for the file.

[0052] In this implementation, the segment allocator generates segment IDs in a monotonically increasing manner and publishes them to the circular buffer, ensuring that all segment IDs stored in the circular buffer are new, unused segments. Thus, when the space pre-allocator allocates logical segments for a file, it also allocates new segments in a monotonically increasing manner, avoiding the repeated allocation of already used logical segments and avoiding the overhead caused by recording space allocation information.

[0053] In the process of publishing and retrieving logical segment IDs, the host-side space pre-allocator acts as a consumer, responsible only for reading free segment IDs from the circular buffer, while the device-side segment allocator acts as a producer, responsible only for adding new segment IDs to the circular buffer. The host-side space pre-allocator and the device-side segment allocator work together to separate the SSD's logical address space from its physical NAND space. This allows the host-side file system to pre-allocate an infinite contiguous space for files, regardless of the actual physical space usage, thereby improving storage efficiency and access speed.

[0054] This embodiment modifies the internal mapping mechanism of the solid-state drive (SSD) and allocates logical segments to files in a monotonically increasing manner, providing an extended logical address space for the file system. In practical applications, the size of the logical space provided by the SSD to the file system through the segment allocator is determined by the number of bits in the logical address. If logical addresses are recorded as integers, a maximum logical address space of 8 ZBytes can be supported in a 64-bit integer system. Based on the actual usage of the SSD, this logical address space will not be exhausted throughout the entire lifespan of the SSD, making it possible for the file system to reserve space for files in segments. Simultaneously, this embodiment uses a global translation table to record mapping relationships, effectively reducing the storage space required for the mapping table and avoiding excessive storage overhead due to the expansion of the logical address space.

[0055] To further improve system access performance, in this embodiment, the device-side segment allocator considers the remaining space of the mapped page when generating new unused segment IDs. Specifically, the segment allocator calculates the free segment IDs and determines the mapped pages in a monotonically increasing manner, including:

[0056] S1: Obtain the segment ID at the end of the circular buffer, increment its value by 1, and use it as a candidate segment ID;

[0057] S2: Calculate the hash value of the candidate segment ID to determine the address of the mapped page to which the candidate segment ID is mapped;

[0058] S3: If the remaining space of the mapped page to which the candidate segment ID is mapped is lower than the average remaining space of the mapped pages, then increment the value of the candidate segment ID by 1 and use it as the new candidate segment ID, and proceed to S2; otherwise, use the current candidate segment ID as the calculation result and its hash value as the address of the mapped page.

[0059] by Figure 1 For example, the process of calculating the segment ID is as follows:

[0060] Obtain the segment ID at the end of the circular buffer, increment its value by 1, and obtain the candidate segment ID (e.g., 0x1FF);

[0061] The hash value of the candidate segment ID is 1023. The remaining space of the corresponding mapped page is 2KB, which is lower than the average remaining space of the mapped page. Skip the segment ID and calculate the next candidate segment ID (e.g., 0x200).

[0062] The hash value of the candidate segment ID is calculated to be 1021. The remaining space of the corresponding mapped page is 3.2KB, which is not lower than the average remaining space of the mapped page. The candidate segment ID is used as the result of the segment ID calculation. The mapping relationship between the segment ID and the mapped page is recorded in the global translation table. The segment ID is then published to the tail of the circular buffer.

[0063] The device-side segment allocator calculates and publishes the IDs of free segments in the above manner, ensuring a balance among the mapping pages and improving the access efficiency of the mapping relationships within the mapping pages.

[0064] In this embodiment, the segment allocator is also used to access the global translation table to obtain the address of the mapped page to which the segment is mapped when the file system writes the logical pages in the segment to the solid-state drive, and to record the mapping relationship between the logical page address and the physical page address in the mapped page.

[0065] In summary, this embodiment expands the logical address space of the file system by modifying the internal mapping mechanism of the SSD, thereby preventing logical address space exhaustion throughout the entire lifespan of the SSD. This method allows the file system to pre-allocate a fixed-size free space window for each file, ensuring the continuity of data layout as the file expands subsequently and effectively avoiding file fragmentation. This not only improves the storage efficiency of the SSD but also significantly enhances the stability and performance of the file system, making it highly valuable for applications that require handling large amounts of data and high-concurrency write operations.

[0066] Example 2:

[0067] A read / write method is applied to the storage system for reducing file fragmentation provided in Embodiment 1 above.

[0068] The read / write method provided in this embodiment includes file write operations; the file write operations include:

[0069] W1: Write the file F to be written according to the logical page size. w Divide into multiple file blocks;

[0070] W2: In the logical address space of the file system, it is file F w Allocate a segment ID and write the file blocks sequentially to the corresponding segment until all file blocks are written or the allocated segment is full;

[0071] W3: If the entire file block is written, proceed to W4; otherwise, write the file blocks in the allocated segment to the solid-state drive and then proceed to W2.

[0072] W4: File write operation completed.

[0073] The read / write method provided in this embodiment also includes: file read operations; such as... Figure 3 As shown, file read operations include:

[0074] R1: Get the segment S containing the content of the file to be read. R ;

[0075] R2: Retrieve segment S from the mapping table R The mapped page address:

[0076] R3: Access segment S based on the mapped page address R The mapped page to obtain segment S R The addresses of the physical pages that each logical page in the table maps to;

[0077] R3: Access segment S R The file read operation ends when the logical pages in the file are mapped to the physical pages to retrieve the content to be read.

[0078] Example 3:

[0079] A read / write controller is applied to the storage system for reducing file fragmentation provided in Embodiment 1 above.

[0080] This embodiment includes:

[0081] A computer-readable storage medium for storing computer programs;

[0082] And a processor for reading a computer program stored in a computer-readable storage medium and executing the read / write method provided in Embodiment 2 above.

[0083] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A storage system for reducing file fragmentation, comprising a host and a solid-state drive; characterized in that, The host has a file system created in it; the file system allocates logical address space for files in units of fixed-size segments; a segment includes multiple logical pages; The host also includes a space pre-allocator, which maintains a circular buffer in the host's memory; the circular buffer stores segment IDs. The firmware of the solid-state drive includes a segment allocator; The segment allocator is used to maintain a global translation table; the global translation table is used to record the mapping relationship from segment ID to mapped page address; the mapped page is a flash page in the solid-state drive used to record the logical page address to physical page address within a segment; The segment allocator is also used to calculate the free segment ID in a monotonically increasing manner and determine the mapped page when there is remaining space in the circular buffer, record the mapping relationship in the global translation table, and publish the calculated free segment ID to the tail of the circular buffer. The space pre-allocator is also used to extract the free segment ID of the circular buffer header and allocate it to the file when the file system needs to allocate space for the file. The segment allocator calculates the free segment ID in a monotonically increasing manner and determines the mapped page, including: S1: Obtain the segment ID at the tail of the circular buffer, increment its value by 1, and use it as a candidate segment ID; S2: Calculate the hash value of the candidate segment ID to determine the address of the mapped page to which the candidate segment ID is mapped; S3: If the remaining space of the mapped page to which the candidate segment ID is mapped is lower than the average remaining space of the mapped pages, then increment the value of the candidate segment ID by 1 and use it as the new candidate segment ID, and proceed to S2; otherwise, use the current candidate segment ID as the calculation result and its hash value as the address of the mapped page.

2. The storage system for reducing file fragmentation as described in claim 1, characterized in that, The hash value of a segment ID is the address of the mapped page to which it is mapped.

3. The storage system for reducing file fragmentation as described in claim 1 or 2, characterized in that, The segment allocator is also used to access the global translation table to obtain the address of the mapped page to which the segment is mapped when the file system writes the logical pages in the segment to the solid-state drive, and to record the mapping relationship from the logical page address to the physical page address in the mapped page.

4. The read / write method for a storage system with reduced file fragmentation as described in any one of claims 1 to 3, characterized in that, Includes file write operations; the file write operations include: W1: Write the file F to be written according to the logical page size. w Divide into multiple file blocks; W2: File F in the logical address space of the file system w Allocate a segment ID and write the file blocks sequentially to the corresponding segment until all file blocks are written or the allocated segment is full; W3: If the entire file block is written, proceed to W4; otherwise, write the file blocks in the allocated segment to the solid-state drive, then proceed to W2. W4: The file write operation has ended.

5. The read / write method as described in claim 4, characterized in that, Also includes: File read operation; The file read operation includes: R1: Get the segment S containing the content of the file to be read. R ; R2: Obtain segment S from the global transformation table. R The mapped page address: R3: Access segment S based on the mapped page address R The mapped page to obtain segment S R The addresses of the physical pages that each logical page in the table maps to; R3: Access segment S R The file read operation ends when the logical pages in the file are mapped to the physical pages to obtain the content to be read.

6. The read / write controller of the storage system for reducing file fragmentation as described in claim 1 or 2, characterized in that, include: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in the computer-readable storage medium and executing the read / write method of claim 4 or 5.

Images