Caching method based on solid state disk

Abstract

The invention discloses a caching method based on a solid state disk, and relates to the technical field of distributed storage. The method comprises the following steps of: configuring a solid state disk cache layer on a post storage layer of a GlusterFS storage service according to a solid state disk basic medium; establishing a unique association between a cache layer of the solid state disk and a storage service through GlusterFS volume configuration; initializing file metadata, constructing a two-stage index architecture, and obtaining a data positioning link according to the two-stage index architecture; after an IO request is received, an IO request processing mechanism is started based on the data positioning link, and read-write interaction of data is completed through the solid state disk cache layer. According to the method, the high-speed read-write characteristic of the solid state disk is utilized, the cache layer of the solid state disk is constructed to replace a traditional mechanical hard disk to serve as a high-frequency data storage medium, a two-stage index architecture is matched to achieve accurate and rapid positioning of data, IO request response time is shortened, and the read-write speed of storage service is increased while the low-cost requirement is considered.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, specifically to a caching method based on solid-state drives. Background Technology

[0002] GlusterFS (a distributed file system) boasts excellent horizontal scalability, capable of supporting petabytes of storage capacity and handling thousands of client requests. GlusterFS aggregates physically distributed storage resources, managing data using a single global namespace. Based on a stackable userspace design, GlusterFS delivers superior performance for diverse data workloads and is widely used in large-scale storage scenarios due to its high scalability and availability. Traditional GlusterFS storage methods include using pure solid-state drives or GlusterFS solutions, but these methods have the following drawbacks: When using pure solid-state drives for storage, the cost is much higher than that of mechanical hard drives of the same capacity. However, the GlusterFS solution uses mechanical hard drives as the underlying storage medium, resulting in higher data read and write latency. It is prone to performance bottlenecks when facing high-frequency IO requests, and cannot meet the needs of high-concurrency business. Furthermore, the caching implementation often depends on specific kernel modules or third-party plugins, which leads to version compatibility issues. In addition, the interaction link between the caching function and the POSIX storage layer is lengthy, which is prone to data synchronization anomalies. As a result, the existing GlusterFS storage cannot achieve a balance of low cost, high read and write speed, and low IO request latency. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the technical problem to be solved by this application is: how to improve the read and write speed of storage services and reduce IO request latency in a low-cost manner.

[0004] To achieve the above objectives, embodiments of this application provide a caching method based on a solid-state drive (SSD), the steps of which include: Create a file system on the solid-state drive (SSD) and obtain the underlying SSD media; Based on the solid-state drive (SSD) base media, a solid-state drive (SSD) cache layer is configured on the POSIX storage layer of the GlusterFS storage service. This SSD cache layer is used to store file metadata and the actual file data. After establishing a unique association between the solid-state drive caching layer and the storage service through GlusterFS volume configuration, the IO request directed transmission link is obtained; Initialize file metadata and build a two-level index architecture, then use the two-level index architecture to locate data links; Upon receiving an IO request, the IO request processing mechanism is initiated based on the data location link, and the data read and write interaction is completed through the solid-state drive caching layer.

[0005] In one implementation, the two-level index architecture includes a global file hash table and a data bucket hash table; wherein the global file hash table includes a hash bucket size parameter, a hash table read / write lock, and a hash linked list array.

[0006] In one implementation, the data bucket hash table is used to manage all active data bucket metadata, which is the metadata of the currently associated valid file and is in a state that can be accessed / called by IO requests.

[0007] In one implementation, the data location link obtained based on the two-level index architecture is as follows: Based on the globally unique identifier of the file in the file metadata, the corresponding file metadata is quickly located from the global file hash table. Then, using the file segment list pointer of the file metadata, the corresponding data bucket metadata is associated through the data bucket hash table. Multiple data bucket metadata corresponding to the same file metadata are linked together according to the association relationship to form a data bucket linked list, forming a secondary location link.

[0008] In one embodiment, the two-level index architecture further includes a file data segment mapping, which serves as a mapping entry to establish the correspondence between file logical offsets and the underlying physical storage locations of the storage medium; the storage medium includes a solid-state drive cache layer and a POSIX storage layer.

[0009] In one implementation, when the IO request processing mechanism is initiated based on the data location link, the process for handling read requests includes: Receive read requests that include a globally unique file identifier and a logical file offset; Search for the corresponding file metadata in the global file hash table based on the file's globally unique identifier; Determine whether the data to be read is already cached in the SSD cache layer. If so, locate the corresponding data bucket metadata through file data segment mapping, then locate the storage location of the data to be read and read the data. Otherwise, allocate file data segment mapping and add the corresponding mapping entry in the two-level index architecture, read the data to be read from the POSIX storage layer and fill it into the SSD cache layer.

[0010] In one implementation, when the IO request processing mechanism is initiated based on the data location link, the write request processing flow includes: Receive write requests that include a globally unique file identifier and a logical file offset; Search for the corresponding file metadata in the global file hash table based on the file's globally unique identifier; Determine if the storage area corresponding to the data to be written exists in the solid-state drive cache layer. If it does, locate the corresponding data bucket metadata through file data segment mapping, then locate the storage location of the data to be written and write the data to be written. Otherwise, allocate file data segment mapping and add the corresponding mapping entry in the two-level index architecture, and write the data to be written to the solid-state drive cache layer.

[0011] In one implementation, during the operation of the solid-state drive (SSD) cache layer, buffer management of the SSD cache layer is performed, and the buffer management process is as follows: Preset the threshold for SSD cache layer usage; When the usage of the solid-state drive cache layer reaches the usage threshold, cache space is reclaimed.

[0012] In one implementation, the cache space reclamation process includes: Cold data is identified using the LRU algorithm, sorted by physical storage location, and then written back to the POSIX storage layer in sequence.

[0013] In one implementation, the free storage blocks in the solid-state drive cache layer are managed uniformly by a storage manager, whose management strategy is as follows: Based on the storage manager's free storage block management capabilities, the allocation requests for solid-state drive cache space are divided into different size levels according to preset rules, and an independent doubly linked list is configured for each size level. After receiving a space allocation request, first match the corresponding size level, and then use the doubly linked list corresponding to that size level to search for and match a free storage block.

[0014] Compared with the prior art, the advantages of this application are: By leveraging the high-speed read and write capabilities of solid-state drives (SSDs), a SSD caching layer is built to replace traditional mechanical hard drives as the high-frequency data storage medium. Combined with a two-level index architecture, accurate and fast data location is achieved, effectively shortening IO request response time. This improves the read and write speed of storage services while meeting the requirements of low cost. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a caching method based on a solid-state drive in an embodiment of this application; Figure 2This is a flowchart illustrating the process of making a read request in an embodiment of this application; Figure 3 This is a flowchart illustrating the write request processing in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0020] Reference Figure 1 As shown in the figure, this application embodiment provides a caching method based on a solid-state drive, the steps of which include: S1. Create a file system on the solid-state drive that satisfies the GlusterFS storage service (brick) cache. After the initialization of the cache storage medium is completed, obtain the solid-state drive base medium, so that the solid-state drive has the basic conditions for cache data storage. S2. Configure a solid-state drive (SSD) cache layer on the POSIX storage layer of the GlusterFS storage service based on the SSD base media. This SSD cache layer is used to store file metadata and actual file data to ensure that the SSD cache layer can interact with the POSIX plugin. S3. After establishing a unique association between the solid-state drive cache layer and the storage service through the GlusterFS volume configuration, the IO request directed transmission link is obtained, so that data read and write interactions can be performed according to the directed transmission link after receiving the IO request; S4. Initialize file metadata and build a two-level index architecture, and obtain data location links based on the two-level index architecture; S5. After receiving the IO request, the IO request processing mechanism is started based on the data location link, and the data read and write interaction is completed through the solid-state drive cache layer.

[0021] By constructing a solid-state drive (SSD) caching layer and leveraging the high-speed read / write capabilities of SSDs to replace traditional mechanical hard drives as high-frequency data storage media, coupled with a two-level index architecture, accurate and rapid data location is achieved, effectively shortening IO request response time. This improves the read / write speed of storage services while meeting the need for low cost.

[0022] In one embodiment, the two-level index architecture includes a first-level index: a global file hash table (HashMap) and a second-level index: a data bucket hash table (BHashMap). The global file hash table is a hash linked list array structure used to quickly locate file metadata. It includes a hash bucket size parameter, a hash table read-write lock (hash_lock), and a hash linked list array (fseglist). The hash linked list array is used to resolve hash collisions, and each linked list node points to a file metadata. The data bucket hash table is used to manage the metadata of all active data buckets (ucm_bid_mm). The active data bucket metadata is the metadata of the currently associated valid file and the data bucket that is in a state that can be accessed / called by IO requests. The data bucket metadata has a built-in data block array, which is used to store the specific data blocks in the data bucket and record the physical offset information of each data block, providing a basis for accurate data location.

[0023] Based on this, the data location path obtained according to the two-level index architecture is as follows: Based on the file's globally unique identifier (u64FileID) in the file metadata, the corresponding file metadata (UCFileID_MM) is quickly located from the global file hash table. Then, using the file segment list pointer (fseglist) of the file metadata, the corresponding data bucket metadata (ucm_bid_mm) is associated through the data bucket hash table. Multiple data bucket metadata corresponding to the same file metadata are linked together according to their association relationship to form a data bucket linked list, thus forming a secondary location link.

[0024] The resulting two-level index architecture avoids the redundancy of traversing a single index, enables accurate and fast data location, shortens data query time, reduces IO request response latency, and further improves data access efficiency by centrally managing related data of the same file using data bucket linked lists.

[0025] In one embodiment, file metadata represents a managed file instance, which includes: The globally unique identifier (u64FileID) of a file is a unique identity assigned by the GlusterFS storage service to the metadata of each file it manages; Reference count (refcount) is used to represent the number of threads or modules currently using the file's metadata. When the count is 0, the file's metadata can be safely released to avoid memory leaks. File-level locks (file_lock) are used to synchronize concurrent access to the file's metadata and its corresponding list of data buckets, ensuring data consistency. A reverse pointer (hashmap) points to the parent global file hash table that manages the file's metadata; The file segment list pointer (fseglist) points to the linked list of data buckets.

[0026] In one embodiment, the two-level index architecture also includes a file data segment mapping (UcFileDataSeg_MM), which serves as a mapping entry to establish the correspondence between the file logical offset and the underlying physical storage location of the storage medium; the storage medium includes: a solid-state drive cache layer and a POSIX storage layer (built based on the POSIX standard, which is the storage carrier for uncached data and the underlying support for the solid-state drive cache layer).

[0027] Based on this, when locating data, the underlying physical storage location can be obtained through mapping entries based on the file logical offset. Then, the linked list members are used to link to the file metadata and data bucket linked list of the corresponding file, providing accurate mapping support for IO request data location.

[0028] Therefore, by accurately mapping the file's logical offset to its physical storage location, index positioning and data read / write path construction are achieved, providing direct positioning support for IO requests.

[0029] In one embodiment, reference is made to Figure 2 The I / O request processing mechanism is initiated based on the data location link. The reading request processing flow includes: Receive read requests from clients that include a globally unique file identifier and a logical file offset; Search for the corresponding file metadata in the global file hash table based on the file's globally unique identifier; Determine whether the data to be read is already cached in the SSD cache layer. If so, locate the corresponding data bucket metadata through file data segment mapping, calculate the storage location of the data to be read in the data block array, and read the data to be read. Otherwise, allocate file data segment mapping and add the corresponding mapping entry in the two-level index architecture, read the data to be read from the POSIX storage layer and fill it into the SSD cache layer, and then feed the target data back to the client.

[0030] This enables tiered processing of read requests, allowing for fast data retrieval when the cache is hit and automatic cache filling when the cache is not hit. This shortens the response time of read requests and provides cache support for subsequent similar requests.

[0031] In one embodiment, reference is made to Figure 3 The write request processing flow, initiated based on the data location link, includes: It receives write requests from clients that include a globally unique file identifier and a logical file offset, and triggers a file-level lock to prevent concurrent write conflicts. Search for the corresponding file metadata in the global file hash table based on the file's globally unique identifier; Determine if the storage area corresponding to the data to be written exists in the SSD cache layer. If it does, locate the corresponding data bucket metadata through file data segment mapping, calculate the storage location of the data block array in the data bucket metadata, and write the data to be written. Otherwise, allocate file data segment mapping and add the corresponding mapping entry in the two-level index architecture, write the data to be written to the SSD cache layer, and then return the write operation result.

[0032] This prioritizes writing data to the SSD cache layer, significantly reducing write latency; the asynchronous write-back mechanism neither blocks subsequent IO requests nor fails to ensure data persistence, balancing write performance and data reliability.

[0033] In one embodiment, during the operation of the solid-state drive (SSD) cache layer, buffer management of the SSD cache layer is performed, and the buffer management process is as follows: Preset the threshold for SSD cache layer usage; When the SSD cache layer usage reaches the usage threshold, the LRU algorithm is triggered to reclaim cache space. The cache space reclamation process includes: Cold data is identified using the LRU algorithm (based on preset standards of data access frequency and recent access time, combined with actual data access frequency and recent access time to determine high-frequency and low-frequency access data, high-frequency access data is marked as hot data and low-frequency access data is marked as cold data). Cold data is sorted by physical storage location and written back to the POSIX storage layer in sequence, while the underlying physical storage location information of the corresponding data in the file data segment mapping is updated synchronously. In this way, while obtaining free storage blocks, hot data can be retained.

[0034] Furthermore, free storage blocks are managed uniformly by the storage manager (HoM). This storage manager achieves persistent data storage by associating with the underlying large file (iFileFD) on the solid-state drive. At the same time, it uses a free block list (hol_list) to maintain free storage block information, providing a basis for the reallocation of cache space. In addition, the retention of hot data can improve cache space utilization and read / write efficiency.

[0035] Based on this, the storage manager's management strategy is as follows: Based on the storage manager's free storage block management capabilities, the allocation requests for solid-state drive cache space are divided into different size levels according to preset rules (such as dividing the range from 4KB to 1MB into 4KB intervals), and an independent doubly linked list is configured for each size level. Upon receiving a space allocation request, the system first matches the corresponding size level, then uses the doubly linked list corresponding to that size level to search for and match free storage blocks, thereby achieving fast space allocation and low fragmentation.

[0036] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0037] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0038] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0039] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0040] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0041] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0042] The above are merely specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the scope of the claims.

Claims

1. A caching method based on a solid-state drive, characterized in that, The steps of this method include: Create a file system on the solid-state drive (SSD) and obtain the underlying SSD media; Based on the solid-state drive (SSD) base media, a solid-state drive (SSD) cache layer is configured on the POSIX storage layer of the GlusterFS storage service. This SSD cache layer is used to store file metadata and the actual file data. After establishing a unique association between the solid-state drive caching layer and the storage service through GlusterFS volume configuration, the IO request directed transmission link is obtained; Initialize file metadata and build a two-level index architecture, then use the two-level index architecture to locate data links; Upon receiving an IO request, the IO request processing mechanism is initiated based on the data location link, and the data read and write interaction is completed through the solid-state drive caching layer.

2. The caching method based on a solid-state drive as described in claim 1, characterized in that: The two-level index architecture includes a global file hash table and a data bucket hash table; The global file hash table includes a hash bucket size parameter, a hash table read / write lock, and a hash linked list array.

3. The caching method based on a solid-state drive as described in claim 2, characterized in that: The data bucket hash table is used to manage the metadata of all active data buckets. The active data bucket metadata is the metadata of the currently associated valid file and the data bucket that is in a state that can be accessed / called by IO requests.

4. The caching method based on a solid-state drive as described in claim 2, characterized in that: The two-level index architecture also includes a file data segment mapping, which serves as a mapping entry to establish the correspondence between the file logical offset and the underlying physical storage location of the storage medium. The storage medium includes a solid-state drive cache layer and a POSIX storage layer.

5. A caching method based on a solid-state drive as described in claim 4, characterized in that: When the IO request processing mechanism is initiated based on the data location link, the process for handling read requests includes: Receive read requests that include a globally unique file identifier and a logical file offset; Search for the corresponding file metadata in the global file hash table based on the file's globally unique identifier; Determine whether the data to be read is already cached in the SSD cache layer. If so, locate the corresponding data bucket metadata through file data segment mapping, then locate the storage location of the data to be read and read the data. Otherwise, allocate file data segment mapping and add the corresponding mapping entry in the two-level index architecture, read the data to be read from the POSIX storage layer and fill it into the SSD cache layer.

6. The caching method based on a solid-state drive as described in claim 4, characterized in that: When the IO request processing mechanism is initiated based on the data location link, the write request processing flow includes: Receive write requests that include a globally unique file identifier and a logical file offset; Search for the corresponding file metadata in the global file hash table based on the file's globally unique identifier; Determine if the storage area corresponding to the data to be written exists in the solid-state drive cache layer. If it does, locate the corresponding data bucket metadata through file data segment mapping, then locate the storage location of the data to be written and write the data to be written. Otherwise, allocate file data segment mapping and add the corresponding mapping entry in the two-level index architecture, and write the data to be written to the solid-state drive cache layer.

7. A caching method based on a solid-state drive as described in claim 1, characterized in that: The data location chain obtained based on the two-level index architecture is as follows: Based on the globally unique identifier of the file in the file metadata, the corresponding file metadata is quickly located from the global file hash table. Then, using the file segment list pointer of the file metadata, the corresponding data bucket metadata is associated through the data bucket hash table. Multiple data bucket metadata corresponding to the same file metadata are linked together according to the association relationship to form a data bucket linked list, forming a secondary location link.

8. A caching method based on a solid-state drive as described in claim 1, characterized in that: During the operation of the solid-state drive (SSD) cache layer, buffer management of the SSD cache layer is performed. The buffer management process is as follows: Preset the threshold for SSD cache layer usage; When the usage of the solid-state drive cache layer reaches the usage threshold, cache space is reclaimed.

9. A caching method based on a solid-state drive as described in claim 8, characterized in that: The cache space reclamation process includes: Cold data is identified using the LRU algorithm, sorted by physical storage location, and then written back to the POSIX storage layer in sequence.

10. A caching method based on a solid-state drive as described in claim 1, characterized in that: The free storage blocks in the solid-state drive cache layer are managed uniformly by a storage manager, whose management strategy is as follows: Based on the storage manager's free storage block management capabilities, the allocation requests for solid-state drive cache space are divided into different size levels according to preset rules, and an independent doubly linked list is configured for each size level. After receiving a space allocation request, first match the corresponding size level, and then use the doubly linked list corresponding to that size level to search for and match a free storage block.

Images