Method and system for managing NVMe SSD based on UBS

Abstract

The invention relates to a method and system for managing an NVMe SSD based on a UBS, and the method comprises the steps: receiving a request transmitted by a client through the same working thread; and performing resource management on a plurality of NVMe SSDs managed by the UBS based on the request. Through the method and the device, operations such as reading / writing, resource application and resource release can be directly performed on the NVMe SSD on the bottom layer based on the UBS, so that reading / writing and resource management of the NVMe SSD in a user mode are realized, and a simple and convenient reading / writing interface is provided for the upper layer. Compared with a resource management mode that switching needs to be frequently achieved between a kernel mode and a user mode in the prior art, the time delay is obviously improved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method and system for managing NVMe SSDs based on UBS. Background Art

[0002] In the application scenario of block storage, ultra-high performance requires that the underlying read / write (IO) be written to the disk as quickly as possible. In this regard, in existing block storage, the backend storage node manages resources and performs read and write operations on NVMeSSD through the interface provided by the block device under the file system or operating system. However, in this existing mode, NVMeSSD device management still has a lot of overhead between the business layer from kernel state to user state.

[0003] There is currently no effective solution to the above-mentioned problems in the related art. Summary of the invention

[0004] The present application provides a method and system for managing NVMe SSD based on UBS, so as to solve the problem in the prior art that resource management of NVMe SSD requires system calls between kernel state and user state, resulting in high latency.

[0005] In the first aspect, the present application provides a user-state block management system, wherein the user-state block management system UBS is used to manage multiple NVMe SSDs, each of which includes: a header area, a block metadata area, an identification area, and a main body area; wherein the header area includes a block management system static area and a block management system dynamic area; the block metadata area includes multiple block metadata, each of which includes metadata of a block; the identification area is used to persist cluster metadata and save the mapping relationship between the physical address and the logical address of the cluster; the main body area includes multiple clusters with continuous physical addresses; the identification area includes multiple IDs, and the multiple IDs correspond one-to-one to the clusters in the main body area.

[0006] In a second aspect, the present application provides a method for managing NVMe SSD based on UBS in the first aspect, the method comprising: receiving a request transmitted by a client through the same working thread; and performing resource management on multiple NVMe SSDs managed by UBS based on the request.

[0007] In a third aspect, the present application provides an electronic device comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute the method steps in the second aspect.

[0008] In a fourth aspect, the present application further provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the method steps in the above-mentioned second aspect of the present application.

[0009] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art: the method provided by the embodiment of the present application,

[0010] The bottom layer of each disk in the UBS in the embodiment of the present application lays out the size of the blocks and clusters in the disk and their corresponding IDs, and saves the mapping relationship between the physical address and the logical address of the cluster, as well as the relationship between the block and the cluster. Based on this, the upper-layer business can directly read, write, apply for resources (block or cluster application), release resources (block or cluster release) and other operations on the underlying NVMe SSD based on UBS, realize the reading and writing and resource management of NVMe SSD in user mode, and provide a simple read and write interface for the upper layer. Compared with the resource management method in the prior art that requires frequent switching between kernel mode and user mode, the method for managing NVMe SSD based on UBS provided in the embodiment of the present application has a significant improvement in latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0013] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0014] Figure 1 A schematic diagram of the bottom layer layout of a UBS provided in an embodiment of the present application;

[0015] Figure 2 A schematic diagram of the mapping between the cluster logical address and the physical address provided in the embodiment of the present application;

[0016] Figure 3 A flowchart of a method for managing NVMe SSD based on UBS of the present application provided in an embodiment of the present application;

[0017] Figure 4 It is a schematic diagram of processing requests based on a work thread provided in an embodiment of the present application;

[0018] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0020] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0021] First, the terms appearing in this application are explained accordingly;

[0022] EBS (Elastic Block Store): an elastically scalable distributed block storage service.

[0023] ESSD (Enhanced SSD): is a product form of EBS.

[0024] Data-Master: A module on the ESSD control path that is responsible for all Data Server resource management, replica recovery, and balancing.

[0025] Data-Client: The client that initiates writing to multiple replicas in ESSD.

[0026] Data Server: A module for single-node data IO storage and resource management on the data path of ESSD.

[0027] Block-Server: It is a module that calls the Data-Client interface to initiate IO to the data server where all replicas are located. It is responsible for accepting customer IO, applying for resources, issuing IO, and establishing indexes.

[0028] DBS: is a collective name for data-master / data-client / data-server.

[0029] NVMe (Non Volatile Memory Host Controller Interface Specification): is a logical device interface specification. It is based on the bus transmission protocol specification of the device logical interface and is used to access non-volatile memory media attached through the PCIExpress (PCIe) bus.

[0030] SSD (Solid State Disk or Solid State Drive): a hard disk made of a solid-state electronic storage chip array.

[0031] UBS (User Block System) manages resource allocation and IO processes on the SSD based on block granularity.

[0032] Header: A part of UBS data division, used to describe the basic information of the management system.

[0033] Extent (cluster): The smallest resource application and management unit in UBS. Its size can be set according to needs. For example, the size of Extent is 4 MiB.

[0034] Meta Data: Each 8 bytes describes an Extent.

[0035] Body: consists of 4 MiB granularity Extent.

[0036] ExtentId (cluster ID): used to describe the physical address of the current Extent, calculated by dividing the physical offset by the Extent size.

[0037] Block: A logical block in UBS, which is essentially an ordered ExtentId table.

[0038] SPDK (Storage Performance Development Kit): A complete set of tools and libraries for SSD devices that support the NVMe protocol to enable high-performance, scalable, full-user storage applications, aiming to significantly reduce the software overhead of the storage IO stack and thus improve storage performance.

[0039] Based on this, the embodiment of the present application provides a user state block management system UBS, UBS is used to manage multiple NVMeSSDs, such as Figure 1Each NVMe SSD shown is divided into: a header area (Header), a block metadata area (BlockMetas), an identification area (IDs) and a body area (Body).

[0040] Among them, the header area includes a block management system static area (Static Header) and a block management system dynamic area (Dynamic Header); the block metadata area includes multiple block metadata, each block metadata includes metadata of a block; the identification area is used to persist cluster metadata and save the mapping relationship between the physical address and logical address of the cluster; the main area includes multiple clusters with continuous physical addresses; the identification area includes multiple IDs, and the multiple IDs correspond one-to-one to the clusters in the main area.

[0041] It can be seen that the bottom layer of each disk in the UBS in the embodiment of the present application has laid out the size of the disk blocks and clusters and their corresponding IDs, and saves the mapping relationship between the physical address and logical address of the cluster, as well as the relationship between the block and the cluster. Based on this, the upper-layer business can directly read, write, apply for resources (block or cluster application), release resources (block or cluster release) and other operations on the underlying NVMe SSD based on UBS, realize the reading and writing and resource management of NVMe SSD in user mode, and provide a simple read and write interface for the upper layer. Compared with the resource management method in the prior art that requires frequent switching between kernel mode and user mode, the method for managing NVMe SSD based on UBS provided in the embodiment of the present application has obvious improvement in latency.

[0042] In an embodiment of the present application, the block management system static area includes a static header field, wherein the static header field includes a first field for characterizing the cluster size, a second field for characterizing the disk sector size, a third field for characterizing the compatible protocol version before and after, and a fourth field for characterizing the size of the space occupied by the cluster ID on the current disk. In a specific example, the block management system static area is represented by BlockSystemStaticRegion, and the size of BlockSystemStaticRegion is 4K bytes. Further, the spdk NVMe in the present application can drive the provision of 4k atomic read and write capabilities. In addition, the first field can be represented by extent_size, whose size can be 4M, the second field is represented by sector_size (sector size), and the sector_size size can be 512 bytes; the third field is represented by proto_version, which is used as a compatibility with different disk formats before and after; the fourth field is represented by meta_size, which represents the size of the space occupied by the extent ID area on the current disk.

[0043] In an embodiment of the present application, the block management system dynamic area includes a dynamic header field, wherein the dynamic header field includes a fifth field for representing the total number of clusters in the static area, a sixth field for representing the number of used clusters, a seventh field for representing the number of valid clusters in the used clusters, and an eighth field for representing the total number of blocks on the current disk.

[0044] In the specific example, the block management system dynamic area is represented by BlockSystemDynamicRegion, and its size is 4K; the fifth field is represented by Size:, which can specifically refer to static_region_->extent_size*static_region_->total_extents, that is, the size of the disk. The sixth field is represented by Used, which indicates the number of clusters that have been used*extent_size. The seventh field is represented by Avail, and the value of Avail is dynamic_region_->size-dynamic_region_->used. The eighth field is represented by Blocks, and Blocks indicates the total number of blocks on the current disk.

[0045] In the embodiment of the present application, the block metadata area (BlockMeta) is composed of multiple BlockMeta data, each BlockMeta includes the meta information of a Block (16G or 2G). Further, the data structure in the block metadata area includes the following fields: the ninth field for representing the block ID provided by the caller outside UBS, the tenth field for representing the block ID inside UBS, the eleventh field for representing the timestamp of the current application to create the block, the twelfth field for representing the timestamp of the most recent application for the current block, the release of the current block, or the change of the current block status, the thirteenth field for representing the block type passed down from the upper layer, the fourteenth field for representing the current block status, and the fifteenth field for representing the writing of the current block to the farthest location.

[0046] In the specific example, the ninth field is represented by block_id, which refers to the ID after erasing the upper 12 bits of the block_id requested from DBS. The tenth field is represented by usbs_blockid, which is the internal ID of UBS, which is only used within UBS and is not visible to the outside. The eleventh field is represented by create_time, which refers to the timestamp when the current block was created. The twelfth field is represented by last_update_time, which refers to the timestamp of the most recent application for the current block or the seal of the current block or the change of the current block status. The thirteenth field is represented by Type, which refers to whether the block type passed down from the upper layer is a sequential block (seq block) or a random block (random block). The fourteenth field is represented by Status, which includes normal, sealed, deleted, temporary, readonly and other states. The fifteenth field is represented by size_in_sector, which is used to indicate the farthest location to which this block is written. The unit is 512B and 4MiB is aligned (because the allocation granularity is 4MiB extent).

[0047] In this regard, the size of each block metadata area is 64 bytes; in the metadata area divided by the disk, 64M of space is allocated to the block metadata area; therefore, the block metadata area can represent approximately 1M blocks. If each block is 16G in size, a maximum of 16G*1M=16P of disk space can be represented.

[0048] In addition, blockid can be a 52-bit block ID provided by the caller (it is the block ID seen by the data-server layer, and the difference from the chunk-id of EBS is that the high 12-bit cluster_id field is removed). usbs_blockid is a 52-byte UBS internal ID, maintained by UBS itself, and only visible inside UBS. In addition, the usbs_blockid inside UBS is monotonically increasing, that is, each newly allocated ID will be assigned an ID larger than all previous IDs, so this ID can avoid the block with the same ID being re-applied when the block is deleted. In addition, UBS maintains a current maximum global ID in memory: g_usbs_blockid. The initialization of this ID is to obtain the maximum UBS ID when loading the block system. Each time a new block is allocated, g_usbs_blockid is incremented, and then the incremented value is assigned to the usbs_blockid of the current block, and it will be persisted. Therefore, in this application, whether the process crashes or the process exits abnormally, it will not affect the monotonically increasing nature of g_usbs_blockid.

[0049] The identification area (IDs area) in the embodiment of the present application is used to persist the metadata information of the cluster (Extent) and to save the mapping relationship between the cluster physical address and the logical address. Among them, the IDs area is composed of multiple IDs, each ID size is 64 bits; each ID corresponds to a 4M cluster in the main area (body area), that is, there are as many IDs as there are 4M clusters. Each ID is divided into two areas: a high address (12 bits) and a low address (52 bits), wherein the high 12 bits indicate the physical cluster corresponding to this ID belongs to which cluster in the block, and the low 52 bits are the value of usbs_blockid. Since 12 bits represent a maximum size of 4K, a block has a maximum of 4K clusters, so a block is at most 4K*4M=16G in size; this is why the current ID structure can only represent a maximum of 16G blocks.

[0050] The main area (Body area) in the embodiment of the present application is composed of many 4M clusters with continuous physical addresses; the cluster is the smallest unit of UBS resource allocation and management. It should be noted that the entire physical disk is divided into continuous 4MiB clusters, and the physical address of each cluster is represented by the offset of the cluster on the physical disk divided by 4MiB (currently using int32_t type values ​​to record, so the current layout determines that the maximum supported disk size is 2*32*4MiB=16PB). In addition, the logical address of each cluster is represented by the upper 12 bits of the corresponding ID field; among them, the upper 12 bits indicate the cluster number in the block to which the cluster belongs, and these 12 bits can be used to represent its logical address in the block.

[0051] Based on the underlying layout of the disk in the above embodiment of the present application, the data persisted by UBS in the BlockMeta area in the embodiment of the present application records the mapping relationship between the external block_id and the UBS internal block id; after scanning the BlockMeta area, it is possible to know which blocks there are and the status of the corresponding blocks. In addition, the UBS area information in IDs persists the mapping information of the physical cluster inside the block to the logical address of the block to which it belongs.

[0052] Furthermore, based on the underlying layout of the disk in the above-mentioned embodiment of the present application, when each disk is formatted, a target data structure (SpdkBlockSystem) for recording index and resource information will be stored in the memory, wherein the target data structure includes a sixteenth field (free_block_header_) for representing the location of all free blocks in the block metadata area, and a seventeenth field (free_extent_index_) for representing the location of all free clusters.

[0053] In the specific example, SpdkBlockSystem records all the block information on this disk, which is represented in memory by a mapping from block_id to Block data structure, where the key of blocks_id is blockid and the value is Block class; free_block_header records the location of all free block meta areas (block metadata) in the BlockMeta area, and each location is represented by a uint32_t; free_extent_index records the location of all free clusters in the cluster area, and each location is represented by a uint32_t.

[0054] In the data structure of Block class, blockid_ is the block id; usbs_blockid is the ID maintained internally by UBD. block_meta_index_ is the index position of the block in block meta, which is used to index the block-level metadata; extents_ stores the index position of each cluster of the block on the disk, which is used to index the physical data carried by the block. The bocks_ data structure is initialized by scanning BlockMeta when OpenBlockSystem; among them, extents_ will be constructed after scanning all IDs in the metadata area.

[0055] It should be noted that during reconstruction, the BlockMeta area is scanned first to obtain all block lists and states, and then the ID area is scanned to construct all cluster lists of the block. Figure 2 , the indexing process for reading and writing a certain dbs_block_id at offset is as follows: first use dbs_blockid as key to find BlockMetaTable and find the corresponding usbs_blockid; then use offset / extent_size in the request as subscript to find the ExtentMetaTable corresponding to usbs_blockid and get the physical extent id; finally, use offset%extent_size in the request as offset to the address specified by extent_id to perform IO. It can be seen that in this application, dbs_block_id can be used to find the corresponding usbs_blockid first, and then the physical cluster ID (extent id) can be obtained according to the offset, realizing a fast conversion from logical address to physical address.

[0056] Based on the UBS of the above embodiment of the present application, the embodiment of the present application also provides a method for managing NVMe SSD based on the UBS of the present application, such as Figure 3 As shown, the method of the embodiment of the present application includes:

[0057] Step 301, receiving a request transmitted by a client through the same working thread;

[0058] Step 302: Perform resource management on multiple NVMe SSDs managed by UBS based on the request.

[0059] It can be seen that in the process of Figure 1 After the bottom layer layout of the UBS shown, the UBS in the embodiment of the present application may only need one working thread for reading, writing or resource management, such as Figure 4As shown, in the specific example, all requests are sent to the working thread by queue message delivery, and processed by the working thread. When the request processing is completed, the user-registered callback will be called in the working thread. Since the user-registered callback is called in the UBS working thread, if the user-registered callback takes a long time, it will block UBS from executing read and write.

[0060] In the embodiment of the present application, resource management in UBS refers to operations such as application or release of clusters and application or release of blocks.

[0061] Based on this, the method for managing clusters in NVMe SSD is as follows: after formatting the NVMe SSD based on UBS, the ID including the identification area (IDs) is formatted to zero, so that the field in the ID used to represent the block ID inside the UBS is set to zero; wherein, the value of the block ID (usbs_block_id) representing the inside of the UBS is allocated incrementally starting from 1, and each time a block ID inside the UBS is allocated, the identification area is updated and persisted in the identification area; the cluster corresponding to the block ID (usbs_block_id) inside the UBS being zero is recorded in the free cluster list of the memory, and in the specific example, it is placed in the free_extents_index_ vector field in the SpdkBlockSystems structure.

[0062] Furthermore, the process of applying for a cluster in the embodiment of the present application includes:

[0063] Step 11, when the current block is full of a cluster size (4M), a new cluster is taken from the free cluster list (free_extents_index_vector) and the identifier of the new cluster (extent_id) is obtained;

[0064] Step 12, update the status of the new cluster in the memory, obtain the 4K aligned starting address of the new cluster in the memory space and the 4K aligned address of the new cluster on the current disk (mem_ptr), write the starting address to the 4K aligned address on the disk (disk_addr), and persist the mapping information between the starting address and the 4K aligned address.

[0065] Furthermore, the process of releasing blocks in the NVMe SSD in the embodiment of the present application includes:

[0066] Step 21, search the record of the corresponding block in the block data structure according to the ID of the current block, set the state of the record to the deleted state, and set the block ID (block_id) provided by the caller outside the UBS to invalid (kBlockSystemInvalidBlockid);

[0067] Step 22, persist the metadata (BlockMeta) of the current block, and after persistence is completed, put the index of each cluster under the block into the free cluster index (extent index) list, set the block ID of the cluster to invalid (kBlockSystemInvalidBlockid), and update the field used to represent the number of used clusters (Avail size) and the field used to represent the number of valid clusters in the used clusters (used size);

[0068] Step 23, return the current block to the block metadata area (free_block_header_) and remove it from the block data structure.

[0069] It should be noted that the above step 22 ensures that even if the metadata of the current cluster is not written to the disk, as long as it is allocated again, since the allocation process will make it persistent, its block_id field will definitely point to the new block, and will not point to the block that has been released above. This ensures that the previously released cluster physical space can still be used, and it does not have to wait until the data-server process is restarted again before it can be recycled. In addition, if the cluster recycled in the above step 22 has not been allocated again in the future, its corresponding Extent ID field will be updated to invalid and persisted in the process of loading metadata when the disk is opened next time. The metadata loading process will update all extent ID fields uniformly at one time.

[0070] Furthermore, the process of applying for a block in the NVMe SSD in the embodiment of the present application includes:

[0071] Step 31, obtain the ID of the free block from the block metadata area (i.e., free_block_header_);

[0072] Step 32: When it is determined that the ID of the free block is valid, the information of the free block is updated in the block metadata area, and the ID of the free block is persisted in the block metadata area.

[0073] The updated free block information includes: block_id, usbs_blockid, create_time, last_update_time, type, status, size_in_sector and other information.

[0074] The above describes the application and release of clusters and blocks in this application. Figure 3 As well as the external interface provided in the embodiments of the present application, further details are provided on how to complete reading, writing and resource management based on the external interface and a working thread.

[0075] like Figure 3 As shown, the current DBS host deploys 8*8T NVMe disks; the read and write operations of these 8 disks can be performed through the same UBS thread; BlockSystemController::OpenBlockSystem. The cores required by UBS will be allocated and the spdk thread of UBS will be initialized when it is called for the first time; it should be noted that if there are multiple instances of BlockSystemController, there will be multiple spdk threads of UBS (currently there is only one BlockSystemController instance);

[0076] The caller obtains a physical disk operation handler by calling BlockSystemController::OpenBlockSystem. The operation performed by OpenBlockSystem is actually to load the metadata information of the UBS disk; the loading of metadata information is completed in the UBS worker thread, and OpenBlockSystem is called in the user thread, so the OpenBlockSystem operation is currently a blocking operation. Fortunately, it is currently only called when the data-server process is started. If the user does not want OpenBlockSystem to block the calling thread, asynchronous optimization can be performed. For example, there is such a requirement in the scenario of temporarily taking the disk offline and then putting it online.

[0077] It can be seen that the submission of reads and writes in the embodiment of the present application is passed by the caller to the UBS worker thread through a message. The UBS worker thread submits the operation of the block to the physical disk for processing through the user-mode NVMe driver of spdk. In addition, the harvesting of reads and writes is performed by the worker thread, and the UBS worker thread harvests the requests that have been completed by the underlying driver through the registered poller. It should be noted that UBS currently has only one worker thread, and according to the current deployment of DBS, one worker thread manages 8 NVMe disks. If the processing slows down on the disk, it will cause the interval between submission and harvesting of the request on the current disk to become longer, but because the spdk NVMe interface is an asynchronous interface, it actually only operates registers such as PCIe, and there will be no blocking, and it will not affect the read and write submission and harvesting of other disks at the UBS layer.

[0078] The external interfaces provided by UBS in the embodiments of the present application are introduced one by one below.

[0079] In the embodiment of the present application, when the first interface (BlockSystemController::OpenBlockSystem) for opening the disk is requested to be called, the resource management method for multiple NVMe SSDs managed by UBS based on the request involved in the above step 302 may further include:

[0080] Step 1-11, allocating a new SpdkBlockSystem in the memory to store metadata of the disk to be opened, wherein, when the first interface is called for the first time, obtaining a core from the configured cores to create a working thread on the core;

[0081] Step 1-12, load the valid UBS external caller provided block ID (block_id) and UBS internal block ID (usbs_blockid) on SpdkBlockSystem;

[0082] Step 1-13, when loading the cluster ID, determine whether the block ID corresponding to the cluster ID is in the block ID (block_id) provided by the caller outside UBS;

[0083] Step 1-14, if the block ID corresponding to the cluster ID is not in the block ID provided by the caller outside UBS, and the block ID corresponding to the cluster ID is not zero, then the block ID corresponding to the memory is set to invalid, and the cluster is added to the free cluster list;

[0084] Step 1-15, if the block ID corresponding to the cluster ID is not in the block ID provided by the caller outside UBS, and the block ID corresponding to the cluster ID is zero, then add the cluster to the free cluster list;

[0085] Step 1-16: If the block ID corresponding to the cluster ID is in the block ID provided by the caller outside the UBS, it is determined that the cluster belongs to the block corresponding to the block ID provided by the caller outside the UBS.

[0086] Repeat the above steps for the next cluster until all clusters are processed. In addition, if the extent_meta flag needs to be updated to true, the new cluster metadata (extent meta) is written to the disk once.

[0087] It should be noted that the block meta / IDs meta information of UBS is loaded from the raw disk into the SpdkBlockSystem object; the operation here is actually a synchronous operation, which will block the caller; if the caller does not want to be blocked, the business after loading the metadata can be placed in the callback specified by OpenBlockSystem, so as to achieve the purpose of non-blocking.

[0088] In the embodiment of the present application, when requesting to call the second interface (BlockSystemController::CloseBlockSystem) for closing the disk, the resource management method for multiple NVMe SSDs managed by UBS based on the request involved in the above step 302 may further include:

[0089] Step 2-11, when there are unfinished read and write requests on the current disk, the unfinished read and write requests are placed in a pending request queue;

[0090] Step 2-12: when the pending request queue meets the conditions for closing the disk, close the current disk.

[0091] In this regard, the BSAction::EXPORT_BLOCK_SYSTEM message is passed through the queue to notify the UBS working thread to close the opened disk. After receiving BSAction::EXPORT_BLOCK_SYSTEM, UBS determines whether there are any unfinished read and write requests. If so, the request is first placed in the pending request queue of UBS. Later (the task driven by the background cycle) will check again from the pending request queue whether the relevant conditions meet the conditions for closing. If they are met, the registered poller will be removed; then the disk controller will be released; finally, the callback registered by the user will be called to notify the user that the disk unloading is complete.

[0092] In the embodiment of the present application, when a third interface (BlockSystem::WriteBlock) for writing data is requested to be called, the read / write operation on multiple NVMe SSDs managed by UBS based on the request involved in the above step 302, so as to manage resources of multiple NVMe SSDs, may further include:

[0093] Step 3-11, execute the asynchronous work function (AsyncWork) and call the data writing function (call the DoWriteBlock api in SpdkBlockSystem::Run to write data);

[0094] It should be noted that when executing steps 3-11, you need to call the SpdkBlockSystem::WriteBlock api, which does some parameter checking and then sends a message to the UBS worker thread for write operations.

[0095] Step 3-12, determining whether writing is completed according to the number of sectors currently written and the number of sectors to be written;

[0096] For this step, whether the writing is completed is determined based on the number of sectors that have been written and the number of sectors that need to be written: ctx->finished_count==ctx->count.

[0097] Step 3-13, when writing is not completed, determine the cluster ID to be written according to the starting sector ID of writing and the number of sectors that have been written;

[0098] For this step, the extent id to be written can be calculated based on the starting sector id to be written (ctx->sectorid) and the number of sectors that have been written (ctx->finished_count).

[0099] Step 3-14, when the cluster corresponding to the cluster ID has not been allocated, allocate the cluster and persist the cluster ID;

[0100] Step 3-15: When the number of sectors previously written is equal to the number of sectors to be written, it is determined that the data writing is completed.

[0101] It should be noted that in this application, DoWriteExtent is called to write data, and a recursive call is made after the writing is completed. If ctx->finished_count == ctx->count is equal, the writing is completed; the callback registered by the user is called to notify the caller that the writing is completed.

[0102] In the embodiment of the present application, when requesting to call the fourth interface (BlockSystem::ReadBlock) for reading data, the resource management method for multiple NVMe SSDs managed by UBS based on the request involved in the above step 302 may further include:

[0103] Step 4-11, execute the asynchronous work function (AsyncWork), and call the function that performs data reading (call the DoReadBlock api in SpdkBlockSystem::Run to read data);

[0104] Before this step, you can call the SpdkBlockSystem::ReadBlock api, which does some parameter checking and then sends a message to the UBS worker thread for reading. Based on this, the UBS worker thread gets the message sent by the caller from the queue through the registered poller and executes the function: AsyncWork.

[0105] Step 4-12, determining whether writing is completed according to the number of sectors currently read and the number of sectors to be read;

[0106] For this step, whether the reading is completed can be determined based on the number of sectors that have been read and the number of sectors that need to be read: ctx->finished_count==ctx->count.

[0107] Step 4-13, when the reading is not completed, determine the starting logical block address LBA to be read in the current cluster, and determine the number of sectors to be read in the current cluster, and read data according to the starting LBA and the sectors to be read;

[0108] For this step, that is, if the parameters are legal, calculate the starting LBA to be read, calculate the number of sectors that can be read in the cluster, and call spdk_NVMe_ns_cmd_read to read the data according to the starting LBA and the number of sectors to be read.

[0109] Step 4-14, when the number of sectors currently read is equal to the number of sectors to be read, it is determined that the data reading is completed.

[0110] In the embodiment of the present application, when requesting to call the fifth interface (BlockSystem::AllockBlock) for allocating blocks, the resource management method for multiple NVMe SSDs managed by UBS based on the request involved in the above step 302 may further include:

[0111] Step 5-11, execute the asynchronous work function (AsyncWork), and call the function to apply for a new block (call the DoAllocBlock api in SpdkBlockSystem::Run to apply for a new block);

[0112] Before this step, you can call the SpdkBlockSystem::AllocBlock api, which does some parameter checking and then sends a message to the UBS worker thread to perform the block application operation.

[0113] Step 5-12, parse out the ID (52 bits) of the requested block in UBS;

[0114] Step 5-13, obtain the currently unused blocks, set the metadata of the unused blocks, and persist the metadata.

[0115] After persistence, the consumer's callback can be called to notify that the block allocation is complete.

[0116] In the embodiment of the present application, when requesting to call the sixth interface (BlockSystem::DeleteBlock) for deleting a block, the resource management method for multiple NVMe SSDs managed by UBS based on the request involved in the above step 302 may further include:

[0117] Step 6-11, execute the asynchronous work function (AsyncWork), and call the function to delete the new block (call the DoDeleteBlock api in SpdkBlockSystem::Run to delete the block);

[0118] Before this step, you can call the SpdkBlockSystem::DeleteBlock api, which does some parameter checking and then sends a message to the UBS worker thread to delete the block. Then, the UBS worker thread gets the message sent by the caller from the queue through the registered poller and executes the function: AsyncWork.

[0119] Step 6-12, modify the status in the metadata information of the block to be deleted to a deleted status, and persist the modified metadata information.

[0120] After the persistence is completed, the consumer's callback is called to inform that the block deletion is complete.

[0121] In the embodiment of the present application, when requesting to call the seventh interface (BlockSystem::SealBlock) for releasing a block, the resource management method for multiple NVMe SSDs managed by UBS based on the request involved in the above step 302 may further include:

[0122] Step 7-11, execute the asynchronous work function (AsyncWork) and call the function to release the block (call the DoSealBlock api in SpdkBlockSystem::Run to release the block);

[0123] Before this step, call the SpdkBlockSystem::SealBlock api, which does some parameter checking and then sends a message to the UBS worker thread to release the block.

[0124] Step 7-12, if the current length to be released is greater than the write length of the block, the release fails;

[0125] Specifically can be size_in_sector <ctx->If count is considered illegal, it cannot be released. In this application, only the length of the data written in the block is allowed to be released.

[0126] Step 7-13, if the current length to be released is less than or equal to the write length of the block, modify the metadata information of the block to be released, wherein the modified metadata information refers to setting the block to a released state, and updating the last write time and the block size (size_in_sector);

[0127] Step 7-14, persist the modified metadata information.

[0128] After the persistence is completed, the user's callback is called to inform that the block is released.

[0129] In the embodiment of the present application, when requesting to call the eighth interface (BlockSystem::DescribeBlock) for obtaining block information, the resource management method for multiple NVMe SSDs managed by UBS based on the request involved in the above step 302 may further include:

[0130] Step 8-11, execute the asynchronous work function (AsyncWork), and call the function to obtain block information (call DoDescribeBlock api in SpdkBlockSystem::Run to obtain block information);

[0131] Before this step, you can call SpdkBlockSystem::DescribeBlock api, which does some parameter checking and then sends a message to the UBS worker thread to get the block information. Then, the UBS worker thread gets the message sent by the caller from the queue through the registered poller; execute the function: AsyncWork.

[0132] Step 8-12, get the current information of the block.

[0133] Among them, obtaining the current information of the block is just a simple reading interface, without updating the data, and then calling the callback of the user.

[0134] In the embodiment of the present application, when requesting to call the ninth interface (BlockSystem::DescribeBlockSystem) for obtaining block system information, the resource management method for multiple NVMe SSDs managed by UBS based on the request involved in the above step 302 may further include:

[0135] Steps 9-11, execute the asynchronous work function (AsyncWork), and call the function to obtain the block system information (SpdkBlockSystem::Run calls DoDescribeBlockSystem api to obtain the block system information);

[0136] Before this step, call SpdkBlockSystem::DescribeBlockSystem api, which does some parameter checking and then sends a message to the UBS worker thread to get the information of the block system. Then, the UBS worker thread gets the message sent by the caller from the queue through the registered poller; execute function: AsyncWork);

[0137] Steps 9-12, obtain the current information of the block system.

[0138] Among them, obtaining the current information of the block is just a simple reading interface, without updating the data, and then calling the callback of the user.

[0139] In the embodiment of the present application, when requesting to call the tenth interface (BlockSystem::SetBlockStatus) for setting the block status, the resource management method for multiple NVMe SSDs managed by UBS based on the request involved in the above step 302 may further include:

[0140] Steps 10-11, execute the asynchronous work function and call the function to set the block status (call the DoSetBlockStatus api in SpdkBlockSystem::Run to set the status of the block system);

[0141] Before this step, call SpdkBlockSystem::SetBlockStatus api, which does some parameter checking and then sends a message to the UBS worker thread to get the block status setting. Then the UBS worker thread gets the message sent by the caller from the queue through the registered poller; execute function: AsyncWork;

[0142] Step 10-12, when the block status modification condition is met, modify the status in the block metadata information, and persist the modified metadata information.

[0143] The conditions for modifying the block status are met when the block status is BLOCK_STATUS_NORMAL&BLOCK_STATUS_TEMPORARY_READONLY&BLOCK_STATUS_RECOVERY. In addition, after the persistence is completed, the user's callback is called to inform that the block status setting is completed.

[0144] like Figure 5 As shown, an embodiment of the present application provides an electronic device, including a processor 511, a communication interface 512, a memory 513 and a communication bus 514, wherein the processor 511, the communication interface 512, and the memory 513 communicate with each other through the communication bus 514.

[0145] Memory 513, used for storing computer programs;

[0146] In one embodiment of the present application, the processor 511 is used to execute the program stored in the memory 513 to implement the method for managing NVMe SSD based on UBS of the present application provided by any of the aforementioned method embodiments, and the role it plays is similar and will not be repeated here.

[0147] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for managing NVMe SSD based on UBS of the present application as provided in any of the aforementioned method embodiments are implemented.

[0148] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0150] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0151] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A user-mode block management system, characterized in that: The user state block management system UBS is used to manage multiple NVMe SSDs, each of which is divided into: a header area, a block metadata area, an identification area, and a body area; Among them, the header area includes a block management system static area and a block management system dynamic area; the block metadata area includes multiple block metadata, each of which includes metadata of a block; the identification area is used to persist cluster metadata and save the mapping relationship between the physical address and the logical address of the cluster; the main area includes multiple clusters with continuous physical addresses; the identification area includes multiple IDs, and the multiple IDs correspond one-to-one to the clusters in the main area respectively.

2. The system according to claim 1, characterized in that The block management system static area includes a static header field, wherein the static header field includes a first field for representing a cluster size, a second field for representing a disk sector size, a third field for representing compatible previous and next protocol versions, and a fourth field for representing a size of space occupied by a cluster ID on a current disk; The block management system dynamic area includes a dynamic header field, wherein the dynamic header field includes a fifth field for representing the total number of clusters in the static area, a sixth field for representing the number of used clusters, a seventh field for representing the number of valid clusters in the used clusters, and an eighth field for representing the total number of blocks on the current disk.

3. The system according to claim 1, characterized in that The data structure in the block metadata area includes the following fields: a ninth field for representing the block ID provided by the caller outside the UBS, a tenth field for representing the block ID inside the UBS, an eleventh field for representing the timestamp of the current application for creating a block, a twelfth field for representing the timestamp of the most recent application for the current block, the release of the current block, or the change of the current block status, a thirteenth field for representing the block type passed down from the upper layer, a fourteenth field for representing the current block status, and a fifteenth field for representing the writing of the current block to the farthest location.

4. The system according to claim 1, characterized in that The memory of the UBS stores a target data structure for recording index and resource information, wherein the target data structure includes a sixteenth field for representing the positions of all free blocks in the block metadata area and a seventeenth field for representing the positions of all free clusters.

5. A method for managing NVMe SSD based on UBS according to any one of claims 1 to 4, characterized in that: The method comprises: Receive requests from clients through the same worker thread; Based on the request, resource management is performed on the multiple NVMe SSDs managed by UBS.

6. The method according to claim 5, characterized in that In the case where the request calls the first interface for opening the disk, the resource management of the plurality of NVMe SSDs managed by the UBS based on the request includes: Allocating a new SPDK-based bare disk block system management system in the memory to store metadata of the disk to be opened, wherein, when the first interface is called for the first time, obtaining a core from the configured cores to create the working thread on the core; Loading a valid block ID provided by a caller outside the UBS and a block ID inside the UBS on the SPDK-based bare disk block system management system; When loading the cluster ID, determining whether the block ID corresponding to the cluster ID is in the block ID provided by the caller outside the UBS; If the block ID corresponding to the cluster ID is not in the block ID provided by the caller outside the UBS, and the block ID corresponding to the cluster ID is not zero, the block ID corresponding to the memory is set to invalid, and the cluster is added to the free cluster list; If the block ID corresponding to the cluster ID is not in the block ID provided by the caller outside the UBS, and the block ID corresponding to the cluster ID is zero, then the cluster is added to the free cluster list; If the block ID corresponding to the cluster ID is in the block ID provided by the caller outside the UBS, it is determined that the cluster belongs to the block corresponding to the block ID provided by the caller outside the UBS.

7. The method according to claim 5, characterized in that In the case where the request calls the second interface for shutting down the disk, the resource management of the plurality of NVMe SSDs managed by the UBS based on the request includes: If there are unfinished read and write requests on the current disk, write the unfinished read and write requests into the pending request queue; When the pending request queue meets the condition for closing the disk, the current disk is closed.

8. The method according to claim 5, characterized in that In the case where the request calls the third interface for writing data, the resource management of the plurality of NVMe SSDs managed by the UBS based on the request includes: Execute the asynchronous work function and call the data writing function; Determine whether writing is complete based on the number of sectors currently written and the number of sectors that need to be written; In the case where writing is not completed, the cluster ID to be written is determined according to the starting sector ID of writing and the number of sectors already written; When the cluster corresponding to the cluster ID is not allocated, allocating the unallocated cluster and persisting the corresponding cluster ID; When the number of sectors currently being written is equal to the number of sectors to be written, it is determined that the data writing is completed.

9. The method according to claim 5, characterized in that In the case where the request calls the fourth interface for reading data, the resource management of the plurality of NVMe SSDs managed by the UBS based on the request includes: Execute the asynchronous work function and call the function that performs data reading; Determine whether the reading is completed based on the number of sectors currently read and the number of sectors to be read; In the case where the reading is not completed, determining the starting logical block address LBA of the current cluster to be read, and determining the number of sectors to be read in the current cluster, and performing data reading according to the starting LBA and the sectors to be read; When the number of sectors currently read is equal to the number of sectors to be read, it is determined that the data reading is completed.

10. The method according to claim 5, characterized in that In the case where the request calls the fifth interface for allocating a block, the resource management of the plurality of NVMe SSDs managed by the UBS based on the request includes: Execute the asynchronous work function and call the function to apply for a new block; Parse the ID of the requested block in UBS; A currently unused block is obtained, metadata of the unused block is set, and the metadata is persisted.

11. The method according to claim 5, characterized in that In the case where the request calls the sixth interface for deleting a block, the resource management of the plurality of NVMe SSDs managed by the UBS based on the request includes: Execute the asynchronous work function and call the function that deletes the new block; The state in the metadata information of the block to be deleted is modified to a deletion state, and the modified metadata information is persisted.

12. The method according to claim 5, characterized in that In the case where the request calls the seventh interface for releasing a block, the resource management of the plurality of NVMe SSDs managed by the UBS based on the request includes: Execute the asynchronous work function and call the function that releases the block; If the current length to be released is greater than the write length of the block, the release fails; If the current length to be released is less than or equal to the written length of the block, the metadata information of the block to be released is modified, wherein the modified metadata information refers to setting the block to a released state and updating the last written time and the block size; Persist the modified metadata information.

13. The method according to claim 5, characterized in that In the case where the request calls the eighth interface for obtaining block information, the resource management of the plurality of NVMe SSDs managed by the UBS based on the request includes: Execute the asynchronous work function and call the function to get block information to get the current information of the block.

14. The method according to claim 5, characterized in that In the case where the request calls the ninth interface for obtaining block system information, the resource management of the plurality of NVMe SSDs managed by the UBS based on the request includes: Execute the asynchronous work function and call the function for obtaining block system information to obtain the current information of the block system.

15. The method according to claim 5, characterized in that In the case where the request calls the tenth interface for setting the block status, the resource management of the plurality of NVMe SSDs managed by the UBS based on the request includes: Execute the asynchronous work function and call the function that sets the block status; When the block status modification condition is met, the status in the metadata information of the block is modified, and the modified metadata information is persisted.

16. The method according to claim 6, characterized in that When managing the clusters in the NVMe SSD, the method includes: After the NVMe SSD is formatted based on the UBS, the ID including the identification area is formatted to zero, so that the field in the ID used to represent the block ID inside the UBS is set to zero; wherein the value representing the block ID inside the UBS is allocated incrementally starting from 1, and each time a block ID inside the UBS is allocated, the identification area is updated and persisted in the identification area; Record the cluster corresponding to the block ID of zero in the UBS in the free cluster list of the memory; When the current block is full of a cluster size, a new cluster is taken from the free cluster list and an identifier of the new cluster is obtained; Update the status of the new cluster in the memory, obtain the 4K-aligned starting address of the cluster of the information in the memory space and the 4K-aligned address of the new cluster on the current disk, write the starting address into the 4K-aligned address on the disk, and persist the mapping information between the starting address and the 4K-aligned address.

17. The method according to claim 5, characterized in that When releasing the block in the NVMe SSD, the method further includes: Searching for a record of a corresponding block in the data structure of the block according to the ID of the current block, setting the state of the record to a deleted state, and setting the block ID provided by the caller outside the UBS to invalid; Persisting the metadata of the current block, and after persistence is completed, putting the index of each cluster under the block into a free cluster list, setting the block ID of the cluster to invalid, and updating the field used to represent the number of used clusters and the field used to represent the number of valid clusters in the used clusters; The current block is returned to the block metadata area and removed from the data structure of the block.

18. The method according to claim 5, characterized in that When applying for a block in the NVMe SSD, the method further includes: Obtaining the ID of the free block from the block metadata area; When it is determined that the ID of the free block is valid, the information of the free block is updated in the block metadata area, and the ID of the free block is persisted in the block metadata area.

Images