Method for storing SPDK metadata and storage system

Abstract

The embodiment of the invention provides a method for storing SPDK metadata and a storage system, and the method comprises the steps: storing the metadata in a built-in metadata storage mode in a compression disk where a compression function of a storage performance development kit SPDK is located; under the condition that any read request of a user is received, the storage position of the corresponding metadata is determined in the metadata storage space according to the carried first parameter information, and the storage position of the compressed data is searched in the corresponding metadata according to the storage position of the metadata; under the condition that any write request of the user is received, the corresponding metadata are read according to second parameter information carried by the write request, the storage position of the compressed data is determined and stored according to the metadata, and the corresponding metadata are written in the metadata storage space according to the storage position of the compressed data. Other persistent memories do not need to be used, so that the method is never limited by scenes, and the failure rate is low; metadata is directly written into a memory, real-time solidification is not needed, and CPU occupation is reduced.

Description

Technical Field

[0001] This invention relates to the field of metadata storage, and more specifically to a method and storage system for storing SPDK metadata. Background Technology

[0002] SPDK (Storage Performance Development Kit) is an open-source software library designed to provide high-performance block storage solutions. It allows developers to bypass the traditional operating system software stack and interact directly with the hardware, thereby achieving higher I / O throughput and lower latency. The REDUCE and VBDEV_COMPRESS modules in SPDK provide compression functionality, enabling the creation of compressed disks on thin-film storage devices, thus achieving efficient use of storage resources.

[0003] In the process of developing this invention, the applicant discovered at least the following problems in the prior art:

[0004] SPDK's compression functionality requires additional metadata storage media, with persistent memory being the preferred choice for optimal metadata performance. Alternatively, a file system can be created on a regular disk to store metadata, creating files and mapping them into memory for use.

[0005] While persistent memory is the optimal metadata storage medium for SPDK compression, its use cases are limited. Therefore, currently, a file system based on a regular disk is used to store compressed disk metadata, but this method has the following technical problems:

[0006] Mapping files from the file system into the SPDK process requires physical memory equal to the size of the metadata space, resulting in high memory consumption. Summary of the Invention

[0007] This invention provides a method and storage system for storing SPDK metadata, which can solve the technical problem of excessive memory usage in the prior art when using ordinary disks to build a file system to store metadata.

[0008] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a method for storing SPDK metadata, comprising:

[0009] Metadata is stored in the compression disk containing the compression function of the Storage Performance Development Kit (SPDK). The built-in metadata storage method is used to store metadata. The built-in metadata storage method includes: allocating multiple memory pages with the same size as the storage block space in the compression disk as metadata storage space. The size of the storage block space is determined according to the type of metadata that SPDK needs to store.

[0010] Upon receiving any read request from a user, the storage location of the corresponding metadata is determined in the metadata storage space based on the first parameter information carried by the read request. The storage location of the compressed data is then located within the corresponding metadata based on the storage location of the metadata. The storage location of the compressed data is used to retrieve the compressed data for the user and decompress the compressed data and return it to the user. The metadata is used to record the storage location of the compressed data.

[0011] Upon receiving any write request from a user, the system reads the corresponding metadata based on the second parameter information carried in the write request, determines the storage location of the compressed data to be written based on the read metadata, and stores it. The system then writes the corresponding metadata into the metadata storage space based on the storage location of the compressed data.

[0012] Secondly, embodiments of the present invention provide an SPDK metadata storage system, comprising:

[0013] Metadata storage space is located in the compression disk where the compression function of the Storage Performance Development Kit (SPDK) is located. It is obtained by allocating memory pages of the same size in the compression disk. The size of the storage block space is determined according to the type of metadata that SPDK needs to store. It is used to store metadata in the compression disk using the built-in metadata storage method.

[0014] The metadata reading unit is used to, upon receiving any read request from a user, determine the storage location of the corresponding metadata in the metadata storage space based on the first parameter information carried by the read request, and locate the storage location of compressed data in the corresponding metadata based on the storage location of the metadata. The storage location of the compressed data is used to retrieve the compressed data for the user and decompress the compressed data and return it to the user. The metadata is used to record the storage location of the compressed data.

[0015] The metadata writing unit is used to read the corresponding metadata according to the second parameter information carried by the write request when any write request is received from the user, determine the storage location of the compressed data to be written by the write request according to the read metadata and store it, and write the corresponding metadata in the metadata storage space according to the storage location of the compressed data.

[0016] Thirdly, embodiments of the present invention provide a computer-readable storage medium that stores one or more programs, which, when executed by a computer device, cause the computer device to perform the aforementioned method for storing SPDK compressed disk metadata.

[0017] Fourthly, embodiments of the present invention provide a computer device, comprising:

[0018] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the aforementioned method of storing SPDK compressed disk metadata.

[0019] The above technical solution has the following beneficial effects: a compressed disk is created on Basebdev (a logic within the compression function) in SPDK. When the compressed disk is created and initialized, it is necessary to determine in advance whether a file system for storing metadata exists. This file system is built based on a regular disk. If it exists, the file system can be used directly to store the metadata. If the file system does not exist, the method for storing SPDK metadata in this invention is used to create a compressed disk on basebdev in SPDK, thus obtaining a storage system for SPDK metadata.

[0020] When creating the compressed disk, the first step is to allocate the required metadata space within Basebdev (a logical component of the compression function). The total space occupied by the metadata space is approximately three-thousandths of the Basebdev space. After the Basebdev space is initialized, memory pages and memory page management structures are allocated. Each memory page is, for example, 4KB in size, and a total of 1024 4KB memory pages are allocated to meet the high concurrency requirements of the business. Because each memory page is very small, the metadata storage medium for the SPDK compressed volume (compressed disk) is transferred from external media to basebdev (a logical component of the compression function) within SPDK. No additional media is needed to store the metadata; this is also known as the built-in metadata solution. The business I / O path of the built-in metadata solution is as follows: Figure 3 As shown. No additional media is needed to store metadata, and the failure rate is low.

[0021] It is used for data storage with compression capabilities and is suitable for building metadata storage systems on SPDK servers that do not have persistent memory. It has no requirements for the use scenario and does not require the use of other persistent memory, thus solving the technical problem of the use of other persistent memory being limited by the scenario.

[0022] The compressed disk storage in basebdev of SPDK in this embodiment of the invention stores metadata. When accessing memory, there is no need to perform memory mapping on the file, so the memory consumption is small and the use of a large amount of physical memory is avoided. The metadata is written directly into memory without real-time hardening and does not occupy the CPU for a long time, thus reducing CPU usage. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a method for storing SPDK metadata according to an embodiment of the present invention;

[0025] Figure 2 This is a logical structure diagram of a block device for storing SPDK metadata according to an embodiment of the present invention;

[0026] Figure 3 This is the business I / O path of the built-in metadata storage space in this embodiment of the invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] like Figure 1 As shown, in conjunction with embodiments of the present invention, a method for storing SPDK metadata is provided, comprising:

[0029] S101: Within the compression disk where the compression function of the Storage Performance Development Kit (SPDK) is located, metadata is stored using a built-in metadata storage method; wherein, the built-in metadata storage method includes: allocating multiple memory pages with the same size as the storage block space within the compression disk as metadata storage space, wherein the size of the storage block space is determined according to the type of metadata to be stored by SPDK;

[0030] S102: Upon receiving any read request from the user, determine the storage location of the corresponding metadata in the metadata storage space according to the first parameter information carried by the read request, find the storage location of the compressed data in the corresponding metadata according to the storage location of the metadata, obtain the compressed data according to the storage location of the compressed data, and decompress the compressed data and return it to the user; wherein, the metadata is used to record the storage location of the compressed data.

[0031] S103: Upon receiving any write request from a user, read the corresponding metadata according to the second parameter information carried by the write request, determine the storage location of the compressed data to be written by the write request based on the read metadata and store it, and write the corresponding metadata in the metadata storage space according to the storage location of the compressed data.

[0032] The compressed disk is created on Basebdev (a logic within the compression function) in SPDK. When the compressed disk is created and initialized, it is necessary to determine in advance whether a file system for storing metadata exists. This file system is built based on a regular disk. If it exists, the file system can be used directly to store the metadata. If the file system does not exist, the compressed disk is created on basebdev in SPDK using the method for storing SPDK metadata in this invention example, thus obtaining the storage system for SPDK metadata.

[0033] When creating the compressed disk, the first step is to allocate the required metadata space within Basebdev (a logical component of the compression function). The total space occupied by the metadata space is approximately three-thousandths of the Basebdev space. After the Basebdev space is initialized, memory pages and memory page management structures are allocated. Each memory page is, for example, 4KB in size, and a total of 1024 4KB memory pages are allocated to meet the high concurrency requirements of the business. Because each memory page is very small, the metadata storage medium for the SPDK compressed volume (compressed disk) is transferred from external media to basebdev (a logical component of the compression function) within SPDK. No additional media is needed to store the metadata; this is also known as the built-in metadata solution. The business I / O path of the built-in metadata solution is as follows: Figure 3 As shown. Because no additional media is needed to store metadata, the failure rate is low.

[0034] It is used for data storage with compression capabilities and is suitable for building metadata storage systems on SPDK servers that do not have persistent memory. It has no requirements for the use scenario and does not require the use of other persistent memory, thus solving the technical problem of the use of other persistent memory being limited by the scenario.

[0035] The compressed disk storage in basebdev of SPDK in this embodiment of the invention stores metadata. When accessing memory, there is no need to perform memory mapping on the file, so the memory consumption is small and the use of a large amount of physical memory is avoided. The metadata is written directly into memory without real-time hardening and does not occupy the CPU for a long time, thus reducing CPU usage.

[0036] The performance of the compressed disk meets the needs of various businesses, and the overall storage space utilization rate is improved by 15%.

[0037] Preferably, the size of the storage block space is determined according to the type of metadata that the SPDK needs to store, including:

[0038] Based on the metadata type to be stored by SPDK, a corresponding metadata block is set for each storage block: Superblock block, Logicalmap block, or Chunkmap block. The metadata type includes at least volume attribute information, address mapping information, and data storage location, wherein:

[0039] Superblock blocks are used to store volume attribute information, which includes at least: a unique identifier for the compressed volume within the compressed disk and the size of the storage unit used to store compressed data; the volume attribute information also includes: metadata area layout, read / write granularity (the size of the read / write granularity seen by the user, for example, 4KB, 512 bytes), and compressed block size (the size of the compression granularity of the compression logic, for example, 16kb).

[0040] The Logicalmap block is used to store address mapping information, which includes the mapping information from user addresses to the Chunkmap blocks within the Chunkmap block.

[0041] The Chunkmap within the Chunkmap block is used to record the data storage location. Metadata refers to the metadata of this data, and the data storage location is the storage location of the compressed data after the original data has been compressed.

[0042] Allocate 4KB aligned storage space from the memory pages of Basebdev. When the metadata storage space is initialized, the volume attribute information is persisted to the Superblock. The 4KB metadata blocks containing Logicalmap and Chunkmap are all initialized to 0.

[0043] Because different metadata is managed separately, there is no problem of centralized storage of metadata during large-scale deployment, and the scope of failure impact is small. This avoids the technical problem that "ordinary disks have a certain probability of failure. When the metadata of multiple compressed volumes is concentrated on an ordinary disk, if the ordinary disk fails, the corresponding compressed volume will become unavailable, resulting in a large amount of data loss."

[0044] Preferably, S102: Upon receiving any read request from the user, the storage location of the corresponding metadata is determined based on the first parameter information carried in the read request; the storage location of the compressed data is located within the corresponding metadata based on the storage location of the metadata; this is used to read data for the user based on the storage location of the compressed data, including:

[0045] S102-1: Upon receiving any read request from a user, calculate the storage block number of the metadata within the compressed disk based on the first parameter information carried by the read request; the first parameter information includes offset and length.

[0046] S102-2: Based on the Logicalmap block number and the offset within the storage block corresponding to the storage block number, the Chunkmap number is obtained by reading the offset position within the Logicalmap block corresponding to the Logicalmap block number.

[0047] S102-3: Calculate the Chunkmap block and its offset within the block based on the Chunkmap sequence number, and obtain the Chunkmap by reading the offset position within the block of the Chunkmap block;

[0048] S102-4: Obtain the storage location of the compressed data after compressing the original data within the Chunkmap, retrieve the compressed data from the storage location of the compressed data, and decompress the compressed data and return it to the user.

[0049] Storage blocks are in 16KB segments. Removing the 16KB segment reveals which 16KB segment the data belongs to, known as the Chunkidx or Chunkmap number. If the data spans multiple chunks, it is divided into two sub-requests and processed separately.

[0050] The formula is expressed as chunkidx = offset / 16kb

[0051] Then, obtain the position index of the chunkmap and store it in the Logicalmap block. Read the Logicalmap block; the Logicalmap index is: Logicalmap_Block_sn = chunkidx / (4096 / 8) + SB_count

[0052] Here, 4096 represents the total number of bytes, and the number 8 represents the size of the position index variable for a single chunkmap, which is 8 bytes.

[0053] The element index within the Logicalmap block is: elem_sn = chunkidx % (4096 / 8)

[0054] After reading the Logicalmap block from the block device based on the Logicalmap sequence number, the `elem_sn`th `u64` value within the Logicalmap block is parsed. This yields the Chunkmap position sequence number, which can be called `Chunkmapidx`.

[0055] Once you have obtained Chunkmapidx, you can read the actual Chunkmap and then read the Chunkmap blocks based on the Chunkmap.

[0056] The sequence number of the chunkmap block is: Chunkmap_Block_sn = chunkmapidx / (4096 / 40) + Logicalmap_block_count + SB_count

[0057] Here, 4096 is the total number of bytes, and the number 40 is the size of a single Chunkmap structure, which is 40 bytes.

[0058] The element index within the chunkmap is: elem_sn = chunkmapidx % (4096 / 40)

[0059] By reading the Chunkmap block according to its sequence number and parsing the elem_sn-th chunkmap structure within the Chunkmap block, the storage location of the compressed data for that chunk can be obtained.

[0060] Next, based on the storage location of the compressed data, it is read, decompressed, and returned to the user.

[0061] Each metadata block is relatively small, resulting in fast read speeds.

[0062] Preferably, S103: Upon receiving any write request from a user, the corresponding metadata is read according to the second parameter information carried in the write request; the storage location of the compressed data to be written by the write request is determined and stored according to the read metadata; and the corresponding metadata is written to the metadata storage space according to the storage location of the compressed data, including:

[0063] S103-1: Upon receiving any write request from the user, calculate the storage block number of the metadata within the compressed disk based on the second parameter information carried by the write request; the second parameter information includes offset and length.

[0064] S103-2: Based on the Logicalmap block number and intra-block offset within the storage block corresponding to the storage block number, if the value of Logicalmap at the intra-block offset position within the Logicalmap block corresponding to the Logicalmap block number is empty, confirm that the data write corresponding to the write request is the first write; if the value of Logicalmap at the intra-block offset position within the Logicalmap block corresponding to the Logicalmap block number is not empty, confirm that the data write corresponding to the write request is an update write.

[0065] S103-3: For the case where the data to be written in the write request is the first write, after compressing the data to be written in the write request to form compressed data, and using the free space in the storage unit as the storage location of the compressed data, the storage location is recorded in the free Chunkmap, and the corresponding Chunkmap sequence number is stored in the Logicalmap whose value in the Logicalmap block is empty, as calculated by the user request.

[0066] S103-4: For the case where the data write corresponding to the write request is an update write, the corresponding Chunkmap is read and parsed according to the Chunkmap sequence number recorded on the Logicalmap block. According to the storage location of the compressed data recorded in the Chunkmap, the corresponding compressed data is read. The read compressed data is decompressed to restore the original data. The original data is overwritten with the new data corresponding to the write request. The storage location of the free Chunkmap block and the compressed data obtained by compressing the new data is determined through the resource bitmap. First, the compressed data obtained by compressing the new data is stored in the new storage location in the storage unit. Then, the new storage location of the compressed data obtained by compressing the new data is recorded in the Chunkmap and written to the free Chunkmap on the disk. Finally, the Chunkmap sequence number is written to the Logicalmap block with an empty value in the Logicalmap block in the storage block where the user request is located.

[0067] Specifically, when a write request is received from a user, the offset and length parameters carried in the write request are used to calculate the 16KB chunk on the compressed disk where the user's write location is located. This chunk is called the chunkidx. Based on the chunkidx, information is read from a specific Logicalmap. If the value recorded in the Logicalmap block is -1ULL, it indicates that the location is a new write; otherwise, it indicates an update write. Specifically:

[0068] New Write: For the case where the write request corresponds to the first write operation, the user's data is compressed. The compressed data is then stored in 4KB granularity in free data locations (obtained by scanning the memory data location bitmap). This is the storage location of the compressed data. The data storage location is then recorded in a Chunkmap structure. The Chunkmap structure is then written to a free location within a Chunkmap block on disk (the free location within the Chunkmap is obtained by scanning the memory Chunkmap resource bitmap). Finally, this free location is written to a specific location within a Logicalmap block (i.e., the location previously identified as -1ULL). This forms a storage address mapping information, which includes the mapping information from the user address to the Chunkmap within the Chunkmap block.

[0069] Update Write: This refers to the case where the data write corresponding to the write request is an update write, involving reading, modification, and writing. Therefore, to achieve write aggregation, after the read and modification in the read-modify-write process are completed, the metadata write request is added to the write waiting queue. Metadata writing is only triggered if there are no other requests in the write waiting queue for that metadata block. At this point, the metadata block may already be carrying multiple metadata write requests. This is equivalent to merging multiple metadata writes into one, avoiding duplicate flushing. After the write is completed, each metadata write request in the write waiting queue is called back sequentially.

[0070] Update Write: Information is read from the Logicalmap block based on the volume attribute information within the compression unit sequence number. Using the chunkmap location information (chunkmapidx) recorded on the Logicalmap block, a specific Chunkmap block is read and parsed. The chunkmap at a specific location within the Chunkmap block contains four pointers describing the location of compressed data. Compressed data is read based on these pointers, decompressed to restore the original data, and then overwritten with the user's new data. New free Chunkmap locations and data storage locations are obtained through the resource bitmap. The chunkmap recording the data storage location is then written to the Chunkmap block, and the Chunkmap location information is written to the Logicalmap block. Finally, the old data storage location and old Chunkmap location are reset on the resource bitmap, indicating that they are free for subsequent write requests.

[0071] Preferably, the method for storing SPDK compressed disk metadata further includes:

[0072] S104: A pass-through cache device is set up for the metadata storage space. In the pass-through cache device, a cache unit is set up for each metadata block. Each cache unit is equipped with a read wait queue and a write wait queue, which are used to respectively mount the read request and write request that need to access the metadata block at the same time.

[0073] S105: Before reading any metadata block, first check whether there is a corresponding cache unit in the pass-through cache device. If there is a corresponding cache unit in the pass-through cache device, then directly access the corresponding cache unit.

[0074] S106: If there is no corresponding cache unit in the pass-through cache device, then read the metadata storage space in the compressed disk;

[0075] S107: If, during the process of reading the metadata storage space in the compressed disk, there are read requests or write requests to access the metadata block added to the waiting queue, then after the reading is completed, the read structure will be notified one by one to the waiting queue with the same request, and the read metadata block will be stored in the cache unit in the pass-through cache device.

[0076] During data access, the need to read and write metadata leads to amplified access to the Basebdev compressed disk and frequent read / write operations. Therefore, a pass-through caching device is constructed to mitigate this issue. The pass-through caching device uses each metadata block as a cache unit (one cache granularity, one metadata block as one cache granularity). Each cache unit has a read wait queue and a write wait queue, used to hold read and write requests that need to access that metadata block simultaneously. This ensures mutual exclusion for multiple requests and allows completion callbacks to be executed through queues. Furthermore, each cache block (cache unit) is managed by an LRU eviction mechanism, reclaiming cache blocks (cache units) that have not been accessed for a long time based on resource usage.

[0077] Before reading a metadata block, the system first checks if a corresponding cache unit exists in the pass-through cache. If a cache unit exists, the data is read directly from the cache unit, bypassing backend memory access and reducing latency. If the pass-through cache unit does not exist, the data needs to be read from the backend compressed disk and stored in the cache unit. During the read of the backend compressed disk, subsequent requests to access the same metadata block are added to either a read wait queue or a write read wait queue. Once the read of the backend compressed disk is complete, each request in the wait queue is called back (notified individually). Therefore, multiple reads of the same metadata block only require one disk read, significantly reducing the frequency of accessing the Basebdev compressed disk.

[0078] Metadata persistence has been transformed from the original method of calling synchronous interfaces to flush data, traversing the file system and kernel, to the SPDK standard asynchronous read / write interface. This means that write wait queues and read wait queues are independent. Metadata read / write follows SPDK's native user-space, asynchronous, and polling mechanisms, no longer occupying CPU resources separately, and eliminating CPU synchronization wait overhead. Compared to existing technologies that require "mapped files and calling synchronous interface methods to flush data," this is much more convenient and efficient.

[0079] By employing a pass-through caching device, the high-frequency access to the compressed disk is eliminated, thus resolving the mutual exclusion problem in the case of high-frequency access to a single metadata block. Furthermore, through read wait queues and write wait queues, the cache accelerates reading and merges writing, thereby improving the efficiency of metadata access.

[0080] Preferably, the method for storing SPDK compressed disk metadata further includes: a step of reloading the metadata storage space after the SPDK process restarts or the machine it resides on restarts, specifically including:

[0081] After the SPDK process restarts or the machine restarts, the Superblock block in the compressed disk is read, the volume attribute information is checked, and it is determined whether the built-in metadata storage method is used in SPDK. If the built-in metadata storage method is used, several memory pages need to be allocated from memory as metadata cache and storage blocks for data reading and writing with the disk, and a control structure for managing each memory page needs to be allocated.

[0082] Read the Logicalmap block and Chunkmap block located after the Superblock block, and restore the Chunkmap bitmap and resource bitmap in sequence.

[0083] First, read the Superblock block of the block device (Basebdev), check the volume attribute information (device attributes), and verify whether the meta_builtin flag is set.

[0084] If the verification confirms the existence of the compressed disk in the invention embodiment, then the 4KB memory pages required for metadata processing and the control structure for managing the memory pages are requested. The memory pages are managed one-to-one through the control structure, and functions such as metadata caching are initialized.

[0085] The metadata stored on the Basebdev disk is read and placed into memory pages. Then, according to the start and end positions of the Logicalmap and Chunkmap blocks provided by the Superblock, the metadata is read to restore the resource bitmap of the metadata. The compressed disk is then reloaded, allowing it to recover its state and ensure data consistency. It can then receive and process user read or write requests.

[0086] like Figure 2 As shown, in conjunction with an embodiment of the present invention, a block device for storing SPDK metadata is provided, comprising:

[0087] Metadata storage space 21 is located in the compression disk where the compression function of the Storage Performance Development Kit (SPDK) is located. It is obtained by allocating memory pages of the same size in the compression disk. The size of the storage block space is determined according to the type of metadata that SPDK needs to store. It is used to store metadata in the compression disk using the built-in metadata storage method.

[0088] Metadata reading unit 22 is used to, upon receiving any read request from a user, determine the storage location of the corresponding metadata in the metadata storage space according to the first parameter information carried by the read request, and search for the storage location of compressed data in the corresponding metadata according to the storage location of the metadata. The storage location of the compressed data is used to obtain the compressed data for the user and decompress the compressed data and return it to the user; wherein, the metadata is used to record the storage location of the compressed data.

[0089] Metadata writing unit 23 is used to read the corresponding metadata according to the second parameter information carried by the write request when receiving any write request from the user, determine the storage location of the compressed data to be written by the write request according to the read metadata and store it, and write the corresponding metadata in the metadata storage space according to the storage location of the compressed data.

[0090] The compressed disk is created on Basebdev (a logic within the compression function) in SPDK. When the compressed disk is created and initialized, it is necessary to determine in advance whether a file system for storing metadata exists. This file system is built based on a regular disk. If it exists, the file system can be used directly to store the metadata. If the file system does not exist, the compressed disk is created on basebdev in SPDK using the method for storing SPDK metadata in this invention example, thus obtaining the storage system for SPDK metadata.

[0091] When creating the compressed disk, the first step is to allocate the required metadata space within Basebdev (a logical component of the compression function). The total space occupied by the metadata space is approximately three-thousandths of the Basebdev space. After the Basebdev space is initialized, memory pages and memory page management structures are allocated. Each memory page is, for example, 4KB in size, and a total of 1024 4KB memory pages are allocated to meet the high concurrency requirements of the business. Because each memory page is very small, the metadata storage medium for the SPDK compressed volume (compressed disk) is transferred from external media to basebdev (a logical component of the compression function) within SPDK. No additional media is needed to store metadata, resulting in a low failure rate; this is also known as the built-in metadata solution. The business I / O path of the built-in metadata solution is as follows: Figure 3 As shown. No additional media is needed to store metadata, and the failure rate is low.

[0092] It is used for data storage with compression capabilities and is suitable for building metadata storage systems on SPDK servers that do not have persistent memory. It has no requirements for the use scenario and does not require the use of other persistent memory, thus solving the technical problem of the use of other persistent memory being limited by the scenario.

[0093] The compressed disk storage in basebdev of SPDK in this embodiment of the invention stores metadata. When accessing memory, there is no need to perform memory mapping on the file, so the memory consumption is small and the use of a large amount of physical memory is avoided. The metadata is written directly into memory without real-time hardening and does not occupy the CPU for a long time, thus reducing CPU usage.

[0094] The compressed disk performance meets the needs of various businesses, and the overall storage space utilization is improved by 15%. Preferably, the storage block includes metadata blocks: Superblock blocks, Logicalmap blocks, or Chunkmap blocks, determined according to the type of metadata to be stored by SPDK. The metadata type includes at least volume attribute information, address mapping information, and data storage location, wherein:

[0095] Superblock blocks are used to store volume attribute information, which includes at least: a unique identifier for the compressed volume within the compressed disk and the size of the storage unit used to store compressed data; the volume attribute information also includes: metadata area layout, read / write granularity (the size of the read / write granularity seen by the user, for example, 4KB, 512 bytes), and compressed block size (the size of the compression granularity of the compression logic, for example, 16kb).

[0096] The Logicalmap block is used to store address mapping information, which includes the mapping information from user addresses to the Chunkmap blocks within the Chunkmap block.

[0097] The Chunkmap block contains a Chunkmap that records the data storage location. Metadata refers to the metadata of the data, and the data storage location is the storage location of the compressed data after the original data has been compressed.

[0098] Allocate 4KB aligned storage space from the memory pages of Basebdev. When the metadata storage space is initialized, the volume attribute information is persisted to the Superblock. The 4KB metadata blocks containing Logicalmap and Chunkmap are all initialized to 0.

[0099] Because different metadata is managed separately, there is no problem of centralized storage of metadata during large-scale deployment, and the scope of failure impact is small. This avoids the technical problem that "ordinary disks have a certain probability of failure. When the metadata of multiple compressed volumes is concentrated on an ordinary disk, if the ordinary disk fails, the corresponding compressed volume will become unavailable, resulting in a large amount of data loss."

[0100] The metadata reading unit 22 is specifically used for:

[0101] Upon receiving any read request from a user, the storage block number where the metadata is located within the compressed disk is calculated based on the first parameter information carried in the read request; the first parameter information includes offset and length.

[0102] Based on the Logicalmap block number and its offset within the storage block corresponding to the storage block number, the Chunkmap number is obtained by reading the offset position of the Logicalmap block corresponding to the Logicalmap block number.

[0103] The Chunkmap block and its intra-block offset are calculated based on the Chunkmap sequence number. The Chunkmap is obtained by reading the intra-block offset position of the Chunkmap block.

[0104] The storage location of the compressed data after compressing the original data is obtained within the Chunkmap. The compressed data is then retrieved from the storage location and decompressed before being returned to the user.

[0105] Storage blocks are in 16KB segments. Removing the 16KB segment reveals which 16KB segment the data belongs to, known as the Chunkidx or Chunkmap number. If the data spans multiple chunks, it is divided into two sub-requests and processed separately.

[0106] The formula is expressed as chunkidx = offset / 16kb

[0107] Then, obtain the position index of the chunkmap and store it in the Logicalmap block. Read the Logicalmap block; the Logicalmap index is: Logicalmap_Block_sn = chunkidx / (4096 / 8) + SB_count

[0108] Here, 4096 represents the total number of bytes, and the number 8 represents the size of the position index variable for a single chunkmap, which is 8 bytes.

[0109] The element index within the Logicalmap block is: elem_sn = chunkidx % (4096 / 8)

[0110] After reading the Logicalmap block from the block device based on the Logicalmap sequence number, the `elem_sn`th `u64` value within the Logicalmap block is parsed. This yields the Chunkmap position sequence number, which can be called `Chunkmapidx`.

[0111] Once you have obtained Chunkmapidx, you can read the actual Chunkmap and then read the Chunkmap blocks based on the Chunkmap.

[0112] The sequence number of the chunkmap block is: Chunkmap_Block_sn = chunkmapidx / (4096 / 40) + Logicalmap_block_count + SB_count

[0113] Here, 4096 is the total number of bytes, and the number 40 is the size of a single Chunkmap structure, which is 40 bytes.

[0114] The element index within the chunkmap is: elem_sn = chunkmapidx % (4096 / 40)

[0115] By reading the Chunkmap block according to its sequence number and parsing the elem_sn-th chunkmap structure within the Chunkmap block, the storage location of the compressed data for that chunk can be obtained.

[0116] Next, based on the storage location of the compressed data, it is read, decompressed, and returned to the user.

[0117] Each metadata block is relatively small, resulting in fast read speeds. Preferably, the metadata writing unit 23 is specifically used for:

[0118] The metadata write timing confirmation subunit is used to calculate the storage block number of the metadata in the compressed disk based on the second parameter information carried by the write request when any write request from the user is received; the second parameter information includes offset and length.

[0119] Based on the Logicalmap block number and intra-block offset within the storage block corresponding to the storage block number, if the value of Logicalmap at the intra-block offset position within the Logicalmap block corresponding to the Logicalmap block number is empty, it is confirmed that the data write corresponding to the write request is the first write; if the value of Logicalmap at the intra-block offset position within the Logicalmap block corresponding to the Logicalmap block number is not empty, it is confirmed that the data write corresponding to the write request is an update write.

[0120] The first write subunit is used for the case where the data to be written in the write request is the first write. After compressing the data to be written in the write request to form compressed data, and using the free space in the storage unit as the storage location of the compressed data, the storage location is recorded in the free Chunkmap, and the corresponding Chunkmap sequence number is stored in the Logicalmap whose value in the Logicalmap block is empty, which is calculated by the user request.

[0121] The update write subunit is used to handle the update write case where the data write corresponding to the write request is an update write. It reads and parses the corresponding chunkmap according to the chunkmap sequence number recorded on the logicalmap block. According to the storage location of the compressed data recorded in the chunkmap, it reads the corresponding compressed data, decompresses the read compressed data to restore the original data, and overwrites the original data with the new data corresponding to the write request. It determines the free chunkmap block and the storage location of the compressed data obtained by compressing the new data through the resource bitmap. First, it stores the compressed data obtained by compressing the new data in the new storage location in the storage unit. Then, it records the new storage location of the compressed data obtained by compressing the new data in the chunkmap and writes it to the free chunkmap on the disk. Finally, it writes the chunkmap sequence number to the empty logicalmap in the logicalmap block in the storage block where the user request is located.

[0122] Specifically, when a write request is received from a user, the offset and length parameters carried in the write request are used to calculate the 16KB chunk on the compressed disk where the user's write location is located. This chunk is called the chunkidx. Based on the chunkidx, information is read from a specific Logicalmap. If the value recorded in the Logicalmap block is -1ULL (0xffffffffffffffffff), it indicates that the location is a new write; otherwise, it indicates an update write. Specifically:

[0123] New Write: For the case where the write request corresponds to the first write operation, the user's data is compressed. The compressed data is then stored in 4KB granularity in free data locations (obtained by scanning the memory data location bitmap). This is the storage location of the compressed data. The data storage location is then recorded in a Chunkmap structure. The Chunkmap structure is then written to a free location within a Chunkmap block on disk (the free location within the Chunkmap is obtained by scanning the memory Chunkmap resource bitmap). Finally, this free location is written to a specific location within a Logicalmap block (i.e., the location previously identified as -1ULL). This forms a storage address mapping information, which includes the mapping information from the user address to the Chunkmap within the Chunkmap block.

[0124] Update Write: This refers to the case where the data write corresponding to the write request is an update write, involving reading, modification, and writing. Therefore, to achieve write aggregation, after the read and modification in the read-modify-write process are completed, the metadata write request is added to the write waiting queue. Metadata writing is only triggered if there are no other requests in the write waiting queue for that metadata block. At this point, the metadata block may already be carrying multiple metadata write requests. This is equivalent to merging multiple metadata writes into one, avoiding duplicate flushing. After the write is completed, each metadata write request in the write waiting queue is called back sequentially.

[0125] Update Write: Information is read from the Logicalmap block based on the volume attribute information within the compression unit sequence number. Using the chunkmap location information (chunkmapidx) recorded on the Logicalmap block, a specific Chunkmap block is read and parsed. The chunkmap at a specific location within the Chunkmap block contains four pointers describing the location of compressed data. Compressed data is read based on these pointers, decompressed to restore the original data, and then overwritten with the user's new data. New free Chunkmap locations and data storage locations are obtained through the resource bitmap. The chunkmap recording the data storage location is then written to the Chunkmap block, and the Chunkmap location information is written to the Logicalmap block. Finally, the old data storage location and old Chunkmap location are reset on the resource bitmap, indicating that they are free for subsequent write requests.

[0126] Preferably, the SPDK metadata storage system further includes a pass-through cache device and a metadata synchronization unit, wherein:

[0127] The pass-through caching device is configured for the metadata storage space, and a cache unit is configured for each metadata block. The cache unit is equipped with a read wait queue and a write wait queue, which are used to load read requests and write requests that need to access the metadata block at the same time, respectively.

[0128] The metadata reading unit is also used to check whether there is a corresponding cache unit in the pass-through cache device before reading any metadata block. If there is a corresponding cache unit in the pass-through cache device, the corresponding cache unit is accessed directly. If there is no corresponding cache unit in the pass-through cache device, the metadata storage space in the compressed disk is read.

[0129] The metadata synchronization unit is configured to, if during the process of reading the metadata storage space in the compressed disk, there are read requests to access the metadata block written to the read waiting queue or write requests added to the write waiting queue, then after the reading is completed, notify the same requests in the read waiting queue one by one of the reading results, and at the same time store the read metadata block into the cache unit in the pass-through cache device.

[0130] Preferably, the SPDK metadata storage system further includes:

[0131] The reload unit is used to reload the metadata storage space after the SPDK process restarts or the machine it resides on restarts. Specifically, the reload unit is used for:

[0132] After the SPDK process restarts or the machine restarts, the Superblock block in the compressed disk is read, the volume attribute information is checked, and it is determined whether the built-in metadata storage method is used in SPDK. If the built-in metadata storage method is used, several memory pages need to be allocated from memory as metadata cache and storage blocks for data reading and writing with the disk, and a control structure for managing each memory page needs to be allocated.

[0133] Read the Logicalmap block and Chunkmap block located after the Superblock block, and restore the Chunkmap bitmap and resource bitmap in sequence.

[0134] During data access, the need to read and write metadata leads to amplified access to the Basebdev compressed disk and frequent read / write operations. Therefore, a pass-through caching device is constructed to mitigate this issue. The pass-through caching device uses each metadata block as a cache unit (one cache granularity, one metadata block as one cache granularity). Each cache unit has a read wait queue and a write wait queue, used to hold read and write requests that need to access that metadata block simultaneously. This ensures mutual exclusion for multiple requests and allows completion callbacks to be executed through queues. Furthermore, each cache block (cache unit) is managed by an LRU eviction mechanism, reclaiming cache blocks (cache units) that have not been accessed for a long time based on resource usage.

[0135] Before reading a metadata block, the system first checks if a corresponding cache unit exists in the pass-through cache. If a cache unit exists, the data is read directly from the cache unit, bypassing backend memory access and reducing latency. If the pass-through cache unit does not exist, the data needs to be read from the backend compressed disk and stored in the cache unit. During the read of the backend compressed disk, subsequent requests to access the same metadata block are added to either a read wait queue or a write read wait queue. Once the read of the backend compressed disk is complete, each request in the wait queue is called back (notified individually). Therefore, multiple reads of the same metadata block only require one disk read, significantly reducing the frequency of accessing the Basebdev compressed disk.

[0136] Metadata persistence has been transformed from the original method of calling synchronous interfaces to flush data, traversing the file system and kernel, to the SPDK standard asynchronous read / write interface. This means that write wait queues and read wait queues are independent. Metadata read / write follows SPDK's native user-space, asynchronous, and polling mechanisms, no longer occupying CPU resources separately, and eliminating CPU synchronization wait overhead. Compared to existing technologies that require "mapped files and calling synchronous interface methods to flush data," this is much more convenient and efficient.

[0137] By employing a pass-through caching device, the high-frequency access to the compressed disk is eliminated, thus resolving the mutual exclusion problem in the case of high-frequency access to a single metadata block. Furthermore, through read wait queues and write wait queues, the cache accelerates reading and merges writing, thereby improving the efficiency of metadata access.

[0138] In conjunction with embodiments of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing one or more programs, which, when executed by a computer device, cause the computer device to perform any of the methods described for storing SPDK compressed disk metadata.

[0139] In conjunction with embodiments of the present invention, a computer device is provided, comprising:

[0140] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the methods for storing SPDK compressed disk metadata.

[0141] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0142] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0143] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0144] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0145] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0146] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0147] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0148] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0149] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for storing SPDK metadata, characterized in that, include: Metadata is stored in the compression disk containing the compression function of the Storage Performance Development Kit (SPDK). The built-in metadata storage method is used to store metadata. The built-in metadata storage method includes: allocating multiple memory pages with the same size as the storage block space in the compression disk as metadata storage space. The size of the storage block space is determined according to the type of metadata that SPDK needs to store. Upon receiving any read request from a user, the storage location of the corresponding metadata is determined in the metadata storage space based on the first parameter information carried by the read request. The storage location of the compressed data is then located within the corresponding metadata based on the storage location of the metadata. The storage location of the compressed data is used to retrieve the compressed data for the user and decompress the compressed data and return it to the user. The metadata is used to record the storage location of the compressed data. Upon receiving any write request from a user, the system reads the corresponding metadata based on the second parameter information carried in the write request, determines the storage location of the compressed data to be written based on the read metadata, and stores it. The system then writes the corresponding metadata into the metadata storage space based on the storage location of the compressed data.

2. The method for storing SPDK compressed disk metadata according to claim 1, characterized in that, The size of the storage block space is determined based on the type of metadata that SPDK needs to store, including: Based on the metadata type to be stored by SPDK, a corresponding metadata block is set for each storage block: Superblock block, Logicalmap block, or Chunkmap block. The metadata type includes at least volume attribute information, address mapping information, and data storage location, wherein: Superblock blocks are used to store volume attribute information, which includes at least: a unique identifier for the compressed volume within the compressed disk and the size of the storage unit used to store the compressed data; The Logicalmap block is used to store address mapping information, which includes the mapping information from user addresses to the Chunkmap blocks within the Chunkmap block. The Chunkmap within the Chunkmap block is used to record the data storage location, which is the storage location of the compressed data after the original data has been compressed.

3. The method for storing SPDK compressed disk metadata according to claim 2, characterized in that, Upon receiving any read request from a user, the method of determining the storage location of the corresponding metadata based on the first parameter information carried in the read request, and searching for the storage location of the compressed data within the corresponding metadata based on the storage location of the metadata, for reading data for the user based on the storage location of the compressed data, includes: Upon receiving any read request from a user, the storage block number where the metadata is located within the compressed disk is calculated based on the first parameter information carried in the read request; the first parameter information includes offset and length. Based on the Logicalmap block number and its offset within the storage block corresponding to the storage block number, the Chunkmap number is obtained by reading the offset position of the Logicalmap block corresponding to the Logicalmap block number. The Chunkmap block and its intra-block offset are calculated based on the Chunkmap sequence number. The Chunkmap is obtained by reading the intra-block offset position of the Chunkmap block. The storage location of the compressed data after compressing the original data is obtained within the Chunkmap. The compressed data is then retrieved from the storage location and decompressed before being returned to the user.

4. The method for storing SPDK compressed disk metadata according to claim 2, characterized in that, Upon receiving any write request from a user, the process involves reading the corresponding metadata based on the second parameter information carried in the write request, determining and storing the storage location of the compressed data to be written based on the read metadata, and writing the corresponding metadata into the metadata storage space based on the storage location of the compressed data. This includes: Upon receiving any write request from a user, the storage block number where the metadata is located within the compressed disk is calculated based on the second parameter information carried in the write request; the second parameter information includes offset and length. Based on the Logicalmap block number and intra-block offset within the storage block corresponding to the storage block number, if the value of Logicalmap at the intra-block offset position within the Logicalmap block corresponding to the Logicalmap block number is empty, it is confirmed that the data write corresponding to the write request is the first write; if the value of Logicalmap at the intra-block offset position within the Logicalmap block corresponding to the Logicalmap block number is not empty, it is confirmed that the data write corresponding to the write request is an update write. For the case where the data to be written in the write request is the first write, after compressing the data to be written in the write request to form compressed data, and using the free space in the storage unit as the storage location of the compressed data, the storage location is recorded in the free Chunkmap, and the corresponding Chunkmap sequence number is stored in the Logicalmap whose value in the Logicalmap block is empty, as calculated by the user request. For the case where the data write corresponding to the write request is an update write, the corresponding Chunkmap is read and parsed according to the Chunkmap sequence number recorded on the Logicalmap block. According to the storage location of the compressed data recorded in the Chunkmap, the corresponding compressed data is read, the read compressed data is decompressed to restore the original data, and the original data is overwritten with the new data corresponding to the write request. The storage location of the free Chunkmap block and the compressed data obtained by compressing the new data is determined by the resource bitmap. First, the compressed data obtained by compressing the new data is stored in the new storage location in the storage unit. Then, the new storage location of the compressed data obtained by compressing the new data is recorded in the Chunkmap and written to the free Chunkmap on the disk. Finally, the Chunkmap sequence number is written to the Logicalmap block with an empty value in the Logicalmap block in the storage block where the user request is located.

5. The method for storing SPDK compressed disk metadata according to claim 2, characterized in that, Also includes: A pass-through caching device is set up for the metadata storage space. In the pass-through caching device, a cache unit is set up for each metadata block. Each cache unit is equipped with a read wait queue and a write wait queue, which are used to load read requests and write requests that need to access the metadata block at the same time, respectively. Before reading any metadata block, first check if there is a corresponding cache unit in the pass-through cache device. If there is a corresponding cache unit in the pass-through cache device, then directly access the corresponding cache unit. If there is no corresponding cache unit in the pass-through cache device, then read the metadata storage space in the compressed disk; If, during the process of reading the metadata storage space in the compressed disk, a read request to access the metadata block is written to the read waiting queue or a write request is added to the write waiting queue, then after the reading is completed, the reading result will be notified to the same request in the read waiting queue or the write waiting queue one by one, and the read metadata block will be stored in the cache unit in the pass-through cache device.

6. The method for storing SPDK compressed disk metadata according to claim 2, characterized in that, Also includes: The steps for reloading the metadata storage space after the SPDK process restarts or the machine it resides on restart include: After the SPDK process restarts or the machine restarts, the Superblock block in the compressed disk is read, the volume attribute information is checked, and it is determined whether the built-in metadata storage method is used in SPDK. If the built-in metadata storage method is used, several memory pages need to be allocated from memory as metadata cache and storage blocks for data reading and writing with the disk, and a control structure for managing each memory page needs to be allocated. Read the Logicalmap block and Chunkmap block located after the Superblock block, and restore the Chunkmap bitmap and resource bitmap in sequence.

7. A storage system for SPDK metadata, characterized in that, include: Metadata storage space is located in the compression disk where the compression function of the Storage Performance Development Kit (SPDK) is located. It is obtained by allocating memory pages of the same size in the compression disk. The size of the storage block space is determined according to the type of metadata that SPDK needs to store. It is used to store metadata in the compression disk using the built-in metadata storage method. The metadata reading unit is used to, upon receiving any read request from a user, determine the storage location of the corresponding metadata in the metadata storage space based on the first parameter information carried by the read request, and locate the storage location of compressed data in the corresponding metadata based on the storage location of the metadata. The storage location of the compressed data is used to retrieve the compressed data for the user and decompress the compressed data and return it to the user. The metadata is used to record the storage location of the compressed data. The metadata writing unit is used to read the corresponding metadata according to the second parameter information carried by the write request when any write request is received from the user, determine the storage location of the compressed data to be written by the write request according to the read metadata and store it, and write the corresponding metadata in the metadata storage space according to the storage location of the compressed data.

8. The SPDK metadata storage system according to claim 7, characterized in that, The storage block includes metadata blocks: Superblock blocks, Logicalmap blocks, or Chunkmap blocks, determined according to the type of metadata to be stored by SPDK. The metadata type includes at least volume attribute information, address mapping information, and data storage location, wherein: Superblock blocks are used to store volume attribute information, which includes at least: a unique identifier for the compressed volume within the compressed disk and the size of the storage unit used to store the compressed data; The Logicalmap block is used to store address mapping information, which includes the mapping information from user addresses to the Chunkmap blocks within the Chunkmap block. The Chunkmap block contains a Chunkmap that records the data storage location, which is the storage location of the compressed data after the original data has been compressed.

9. The SPDK metadata storage system according to claim 8, characterized in that, It also includes a pass-through cache unit and a metadata synchronization unit, wherein: The pass-through caching device is configured for the metadata storage space, and a cache unit is configured for each metadata block. The cache unit is equipped with a read wait queue and a write wait queue, which are used to load read requests and write requests that need to access the metadata block at the same time, respectively. The metadata reading unit is also used to check whether there is a corresponding cache unit in the pass-through cache device before reading any metadata block. If there is a corresponding cache unit in the pass-through cache device, the corresponding cache unit is accessed directly. If there is no corresponding cache unit in the pass-through cache device, the metadata storage space in the compressed disk is read. The metadata synchronization unit is configured to, if during the process of reading the metadata storage space in the compressed disk, there are read requests to access the metadata block written to the read waiting queue or write requests added to the write waiting queue, then after the reading is completed, notify the same requests in the read waiting queue one by one of the reading results, and at the same time store the read metadata block into the cache unit in the pass-through cache device.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which, when executed by a computer device, cause the computer device to perform the method of storing SPDK compressed disk metadata as described in any one of claims 1-6.

11. A computer device, characterized in that, include: processor; And a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the method of storing SPDK compressed disk metadata as described in any one of claims 1-6.

Images