Data management method and device of storage system and storage medium
By adopting multi-level storage structure and dynamic hierarchical selection methods in distributed storage systems, the problem of low write data in the prior art is solved, and more efficient data writing operations are achieved.
Patent Information
- Application Number
- CN202411688298.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-02-28
AI Technical Summary
Existing distributed storage systems are difficult to efficiently support random file modification and write operations, and usually only support append write operations, resulting in low data writing efficiency.
A multi-level storage structure is adopted, and the storage space of the corresponding level is selected according to the size of the written data, and the storage space header is updated to identify the target level where the data is located.
By dynamically selecting the appropriate storage hierarchy, the impact of fixed storage space size on writing efficiency is reduced and the efficiency of writing data is improved.
Smart Images
Figure CN119536656A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of distributed storage technology, and in particular to a data management method, device and storage medium of a storage system. Background Art
[0002] Distributed storage systems basically cannot support random modification and writing of files (read, modify and write), and usually only support append operations. In order to support distributed file systems, secondary development of storage bases based on append operations is required. In order to improve the read and write efficiency of the files to be processed, the files will be segmented.
[0003] JuiceFS (open source distributed file system) implements data storage through object storage (append write distributed storage system) at the bottom layer, divides files into different Block objects according to 4MB, and implements a metadata index management by itself. When processing modification and writing, by writing new Block objects, modifying metadata indexes, marking expired old Block objects as invalid data, and subsequently deleting them, the essence is to re-upload new objects to achieve random modification and writing.
[0004] CephFS will split the file by object_size (4MB by default), and each block corresponds to a RADOS object. When processing modification and writing, it is often necessary to read part of the object content first, modify it in memory, and then write it. Summary of the invention
[0005] One object of the present disclosure is to improve the data writing efficiency of a distributed storage system.
[0006] According to one aspect of some embodiments of the present disclosure, a data management method for a storage system is proposed, including: obtaining data to be written; determining an identifier of a first-level storage space to be written according to a writing position of the data to be written and a length of the data to be written; determining a target level to which the storage space to which the data to be written is aligned belongs according to the size of the data to be written, wherein the target level is the first level, the second level or the third level, the storage space of the first level includes at least two second-level storage spaces, and the storage space of the second level includes at least two third-level storage spaces; writing the data to be written into the storage space of the target level corresponding to the identifier of the first-level storage space.
[0007] In some embodiments, the data management method also includes: updating the storage space header corresponding to the first-level storage space to be written, wherein each first-level storage space includes a corresponding storage space header, and the storage space header stores status information of the first-level storage space, the status of the second-level storage space and the status information of the third-level storage space.
[0008] In some embodiments, the first-level storage status information is a validity indicator of data stored in the first-level storage space.
[0009] In some embodiments, the status information of the second-level storage space is a bitmap of the second-level storage space storing valid information in the first-level storage space.
[0010] In some embodiments, the status information of the third-level storage space is a bitmap of the third-level storage space storing valid information in the first-level storage space.
[0011] In some embodiments, determining the identifier of the first-level storage space to be written based on the writing position of the data to be written and the length of the data to be written includes: determining the starting identifier based on the writing position of the data to be written and the size of each first-level storage space; determining the ending identifier based on the starting identifier, the size of each first-level storage space and the length of the data to be written, wherein the identifier of the first-level storage space to be written includes the identifiers of all first-level storage spaces from the starting identifier to the ending identifier, including the starting identifier and the ending identifier.
[0012] In some embodiments, determining the target level to which the storage space to which the data to be written is aligned belongs based on the size of the data to be written includes: determining the starting position and the ending position stored in each first-level storage space to be written based on the writing position of the data to be written, the length of the data to be written, and the size of each first-level storage space; determining the target level to which the storage space to which the data to be written is aligned in each first-level storage space to be written belongs based on the starting position and the ending position, as well as the sizes of the first-level, second-level, and third-level storage spaces.
[0013] In some embodiments, determining the starting position and ending position stored in each first-level storage space to be written includes: determining the starting position and ending position in the storage space corresponding to the start identifier according to the writing position of the data to be written and the size of each first-level storage space; determining the ending position in the storage space corresponding to the end identifier according to the writing position of the data to be written, the length of the data to be written, the end identifier and the size of each first-level storage space.
[0014] In some embodiments, determining the starting position and the ending position stored in each first-level storage space to be written also includes: when the difference between the ending identifier and the starting identifier is greater than 1, in the first-level storage space corresponding to the identifier between the starting identifier and the ending identifier, the starting position is 0, and the ending position is the size of the first-level storage space minus 1; determining the target level to which the storage space aligned with each first-level storage space to be written belongs also includes: when the difference between the ending identifier and the starting identifier is greater than 1, in the first-level storage space corresponding to the identifier between the starting identifier and the ending identifier, the target level is the first level.
[0015] In some embodiments, updating the storage space header corresponding to the first-level storage space to be written includes at least one of the following: when the target level is the first level, updating the status information of the first-level storage space to be valid, and updating the status information of the second-level and third-level storage spaces to be invalid; when the target level is the second level, updating the location of the data to be written in the bitmap of the second-level storage space to be valid, and updating the status information of the third-level storage space to be invalid; when the target level is the third level, updating the location of the data to be written in the bitmap of the third-level storage space to be valid.
[0016] In some embodiments, the data management method also includes: obtaining the position of the data to be read and the length of the data to be read; determining the identifier of the first-level storage space to be read based on the position of the data to be read and the length of the data to be read; and reading data of at least one level of storage space in the corresponding storage space based on the identifier of the first-level storage space to be read.
[0017] In some embodiments, reading data of at least one level of storage space in the corresponding storage space includes: reading data of a storage space of a level identified as valid according to a storage space header corresponding to a first level of storage space to be read.
[0018] In some embodiments, according to the storage space header corresponding to the first-level storage space to be read, reading the data of the storage space of the level identified as valid includes: when the status information of the third-level storage space includes an identification that valid information is stored, reading the data identified as valid information in the third-level storage space; when the status information of the second-level storage space includes an identification that valid information is stored, reading the data identified as valid information in the second-level storage space; when the status information of the first-level storage space includes an identification that valid information is stored, reading the data of the third-level storage space.
[0019] In some embodiments, the reading priority of data in the third-tier storage space is higher than the reading priority of data in the second-tier storage space, which is higher than the reading priority of data in the first-tier storage space.
[0020] In some embodiments, the first level is the chunk, the second level is the block, and the third level is the page.
[0021] According to one aspect of some embodiments of the present disclosure, a data management device for a storage system is proposed, comprising: a data acquisition unit, configured to acquire data to be written; an identification determination unit, configured to determine an identification of a first-level storage space to be written according to a writing position of the data to be written and a length of the data to be written; a level determination unit, configured to determine a target level to which the storage space to which the data to be written is aligned belongs according to a size of the data to be written, wherein the target level is the first level, the second level or the third level, the storage space of the first level includes at least two storage spaces of the second level, and the storage space of the second level includes at least two storage spaces of the third level; and a writing unit, configured to write the data to be written into the storage space of the target level corresponding to the identification of the storage space of the first level.
[0022] In some embodiments, the data management device also includes: an update unit, configured to update the storage space header corresponding to the first-level storage space to be written, wherein each first-level storage space includes a corresponding storage space header, and the storage space header stores status information of the first-level storage space, the status of the second-level storage space and the status information of the third-level storage space.
[0023] In some embodiments, the data management device also includes: a reading unit, configured to obtain a reading position of the data to be read and a length of the data to be read, and read data of at least one level of storage space in the corresponding storage space based on an identifier of the first level of storage space to be read, wherein the identifier determination unit is further configured to determine the identifier of the first level of storage space to be read based on the reading position of the data to be read and the length of the data to be read.
[0024] According to one aspect of some embodiments of the present disclosure, a data management device for a storage system is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the above data management methods based on instructions stored in the memory.
[0025] According to one aspect of some embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, any one of the above data management methods is implemented.
[0026] According to one aspect of some embodiments of the present disclosure, a computer program product is provided, comprising computer instructions, and when the computer instructions are executed by a processor, any one of the above data management methods is implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The drawings described herein are used to provide further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute improper limitations on the present disclosure.
[0028] Figure 1 The present invention provides a flowchart of some embodiments of a data writing process in a data management method of a storage system of the present invention.
[0029] Figure 2 Schematic diagram of some embodiments of a storage space header in a data management method of a storage system disclosed in the present invention.
[0030] Figure 3 The present invention is a flowchart of some embodiments of updating a storage space header in a data management method of a storage system of the present invention.
[0031] Figure 4 The present invention provides a flowchart of some embodiments of a data reading process in a data management method of a storage system of the present invention.
[0032] Figure 5 Flowcharts of other embodiments of the data reading process in the data management method of the storage system disclosed in the present invention.
[0033] Figure 6 Schematic diagrams of some embodiments of the data management device of the storage system disclosed in the present invention.
[0034] Figure 7 Schematic diagrams of other embodiments of the data management device of the storage system disclosed in the present invention.
[0035] Figure 8 Schematic diagrams of some further embodiments of the data management device of the storage system disclosed in the present invention. DETAILED DESCRIPTION
[0036] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments.
[0037] In the related technology, data is divided into chunks of fixed size, which is simple and easy to design and implement, has a clear structure, and has good write performance.
[0038] However, the inventors found that such a design is not flexible enough, and different problems will arise if the Chunk is too small or too large. If the Chunk is too small, it is not friendly to the reading and writing of large blocks of files, and many Chunks will be read and written each time. For example, the Chunk size is 4KB, and the reading and writing are both 1MB. In this way, each read and write request must read 256 Chunks, affecting the bandwidth of large-block reading and writing. If the Chunk is too large, it is not friendly to the reading and writing of small blocks of files, and there is an obvious read-write amplification problem, which affects the IOPS of small-block reading and writing. For example, if the Chunk size is 1MB, writing 4KB data requires writing 1MB of data. When overwriting 4KB data, it is necessary to read 1MB of data first, then update and write 1MB, and the read and write are amplified by 256 times.
[0039] In response to the above problems, the present disclosure proposes a data management method, device and storage medium for a storage system, which adopts a multi-level storage structure and selects the storage space of the corresponding level according to the size of the write data, thereby reducing the impact of the fixed size of the set storage space on the write efficiency and improving the efficiency of writing data.
[0040] The flowcharts of some embodiments of the data writing process in the data management method of the storage system disclosed in the present invention are as follows: Figure 1 shown.
[0041] In step S11, data to be written is obtained.
[0042] When the user needs to write data, the user request passes through the system call, passes through the kernel VFS (Virtual FileSystem, virtual file system layer), and passes the request to the kernel Fuse module. The kernel Fuse module then passes the request to the user-mode UFS-Client (a user-mode module developed based on Libfuse). According to the request type, the UFS-Client sends the data read and write request to the UFS-Data-Server (data storage node) and the metadata request to the UFS-Meta-Server (metadata storage node). The method for obtaining the data to be written can adopt any method in the relevant technology. In some embodiments, in addition to the data itself, the write location can also be obtained, that is, the length of the data can be determined.
[0043] In step S12, the identifier of the first-level storage space to be written is determined according to the writing position of the data to be written and the length of the data to be written.
[0044] In some embodiments, the method for obtaining the writing position (offset) of the data to be written and the length (len) of the data to be written may be based on any method in the related art.
[0045] In some embodiments, if the length of the data to be written is less than or equal to the size of a first-level storage space, that is, all the data to be written can be stored by a first-level storage space, then the identifier of the first-level storage space to be written = offset / the size of the first-level storage space.
[0046] In some embodiments, if the length of the data to be written is greater than the size of a first-level storage space, that is, it is difficult to store all the data to be written using a first-level storage space, then the start identifier can be determined according to the location where the data to be written is written and the size of each first-level storage space, and the end identifier can be determined according to the start identifier, the size of each first-level storage space and the length of the data to be written, then the identifier of the first-level storage space to be written is [start identifier, end identifier]. For example, taking the first level as chunk as an example, the identifier Chunkid of the first-level storage space to be written involved in a write request is calculated as follows:
[0047] start_chunkid=offset / chunk_size
[0048] end_chunkid=(offset+len–1) / chunk_size
[0049] Then Chunkid is [start_chunkid,end_chunkid]
[0050] In the above formula, start_chunkid is the starting identifier, end_chunkid is the ending identifier, and chunk_size is the size of the first-level storage space.
[0051] In step S13, the target level of the storage space to which the data to be written is aligned is determined according to the size of the data to be written. In the embodiment of the present disclosure, the levels of the storage space are divided into the first level, the second level and the third level, and the target level is the first level, the second level or the third level.
[0052] The storage space of the first level is larger than the storage space of the second level, for example, it includes at least two storage spaces of the second level; the storage space of the second level is larger than the storage space of the third level, for example, it includes at least two storage spaces of the third level.
[0053] In some embodiments, the first level is Chunk, the second level is Block, and the third level is Page. The level names and forms in the above embodiments are examples of feasible methods and do not constitute improper limitations on the present disclosure.
[0054] For the convenience of description in some embodiments below, the first level is Chunk, the second level is Block, and the third level is Page as an example. Other first, second, and third level forms are also applicable to various embodiments.
[0055] In some embodiments, the number of blocks in each Chunk is chunk_size / block_size, and the number of pages in each Block is block_size / page_size, where block_size is the size of the second-level storage space, and page_size is the size of the third-level storage space. In some embodiments, page_size is 4KB.
[0056] In some embodiments, if the data to be written is aligned with the Chunk size, it is saved as Chunk data; if the data size to be written is smaller than the Chunk size and Block aligned, it is saved as Block data, for example, it can be stored in one or more Blocks; if the data size to be written is smaller than the Block size, it is saved as Page data, for example, it can be stored in one or more Pages.
[0057] In some embodiments, when the data to be written needs to occupy multiple first-level storage spaces, it is necessary to determine the target level for each first-level storage space occupied. For example, the starting position and the ending position of the data to be written in each first-level storage space to be written can be determined based on the writing position of the data to be written, the length of the data to be written, and the size of each first-level storage space, and then the target level to which the data to be written in each first-level storage space to be written belongs can be determined based on the starting position and the ending position, as well as the sizes of the first-level, second-level, and third-level storage spaces.
[0058] For the first occupied first-level storage space and the last occupied first-level storage space, there may be a situation where the entire first-level storage space cannot be occupied, and the target level may be any one of the first, second, and third levels. In some embodiments, to determine the starting position and the ending position in the storage space corresponding to the starting identifier, the position where the data to be written is written and the size of each first-level storage space can be referenced. To determine the starting position and the ending position in the storage space corresponding to the ending identifier, the position where the data to be written is written, the length of the data to be written, the ending identifier, and the size of each first-level storage space can be referenced. Furthermore, the target level is determined based on the size of the storage space between the starting position and the ending position, and the size of the storage space of each level.
[0059] In addition, if the number of first-level storage spaces occupied is greater than 2, then except for the first first-level storage space and the last first-level storage space, the other first-level storage spaces will be full, that is, the starting position is 0, and the end position is the size of the first-level storage space minus 1. In this case, the target level is the first level, for example, the data is Chunk-aligned, so it is only necessary to determine the target level for the data to be written stored in the first and last first-level storage spaces, without confirming the target level one by one, which improves processing efficiency.
[0060] For example, for the first level of storage space start_chunk occupied:
[0061] chunk_offset = offset % chunk_size
[0062] chunk_len = min(len, chunk_size – chunk_offset)
[0063] For the last first-level storage space occupied by end_chunk:
[0064] chunk_offset = 0
[0065] chunk_len = (offset + len) – end_chunkid * chunk_size
[0066] If end_chunk-start_chunk ≥ 2, then for each first-level storage space in the middle:
[0067] chunk_offset = 0
[0068] chunk_len = chunk_size
[0069] Among them, chunk_offset is the relative starting position in the first level, % is the remainder calculation, min() is the minimum value operation, and chunk_len is the size of the space occupied in the storage space of the first level.
[0070] In step S14, the data to be written is written into the storage space of the target level corresponding to the identifier of the storage space of the first level.
[0071] Based on the method in the embodiment shown above, through a multi-layer storage space data organization model, it is possible to select a suitable target level during the data writing process, reduce the amplification factor of small block write requests, and also reduce the number of write units involved in large block write requests, thereby improving write efficiency.
[0072] In some embodiments, Figure 1 As shown, the data writing process in the data management method of the storage system disclosed in the present invention further includes step S15. Step S15 and step S14 can be executed in parallel, or step S15 can be executed first after step S13, and then step S14.
[0073] In step S15, the storage space header corresponding to the first-level storage space to be written is updated. In some embodiments, each first-level storage space includes a corresponding storage space header, such as chunk_head. The storage space header stores the status information of the first-level storage space, such as whether the data currently stored in the first-level storage space is valid or invalid. The storage space header also stores the status information of the second-level storage space and the third-level storage space, which can be stored in the form of a bitmap, such as Figure 2 As shown in . Taking the first level as Chunk, the second level as Block, and the third level as Page as an example, the state information of the storage space of the first level is chunk_state, the state information of the storage space of the second level is block_bit_map, and the state information of the storage space of the third level is page_bit_map.
[0074] In some embodiments, when updating the storage space header, if the target tier is the first tier, the status information of the storage space of the first tier is updated to be valid, and the status information of the storage space of the second tier and the third tier is updated to be invalid.
[0075] In some embodiments, when updating the storage space header, if the target level is the second level, the location of the data to be written in the bitmap of the second level storage space is updated to valid, and the status information of the third level storage space is updated to invalid.
[0076] In some embodiments, when updating the storage space header, if the target level is the third level, then in the case where the target level is the third level, the location where the data to be written is stored in the bitmap of the storage space of the third level is updated to be valid.
[0077] In some embodiments, taking the first level as Chunk, the second level as Block, and the third level as Page as an example, the process of updating the storage space header can be as follows: Figure 3 As shown, in this embodiment, the corresponding Figure 1In the illustrated embodiment, step S15 is performed first, and then step S14 is performed, wherein step 356 is an embodiment of the above-mentioned step S14.
[0078] In step 351, the information in chunk_head is read.
[0079] In step 352 , it is determined whether the target level is Chunk. If the target level is Chunk, step 353 is executed; otherwise, step 354 is executed.
[0080] In step 353, chunk_state is set to 1, and block_bit_map and page_bit_map are all set to 0. In the current embodiment, 1 represents valid and 0 represents invalid. In some embodiments, 0 can be used to represent valid and 1 can represent invalid, then chunk_state is set to 0, and block_bit_map and page_bit_map are all set to 1. For this embodiment, those skilled in the art can adjust it according to their understanding, and the following description of this disclosure will not be repeated.
[0081] Furthermore, the data written to the chunk will overwrite the data in the original block and page, so all block and page data in the chunkid are marked for deletion, and the data in the chunk is taken as a whole.
[0082] In step 354, when the target level is a block, the corresponding position of the block in the bitmap where the data to be written is set to 1, and the corresponding position of the page contained in the block in the bitmap is set to 0, and the data in the page contained in the block is marked for deletion, with the data in the block as a whole.
[0083] In step 355 , when the target level is a page, the corresponding position of the page where the data to be written is written in the bitmap is set to 1.
[0084] Further, step 356 may be executed to perform an atomic write operation to ensure data consistency and integrity and complete the update of chunk_head and data in the chunk, block or page.
[0085] Based on the method in the above embodiment, the target level occupied by the data stored in the storage space can be identified through the storage space header, and the occupancy status of the space at each level in the storage space can be updated, thereby facilitating the search for the correct data to be read when reading data and improving the reliability of the reading operation.
[0086] The flowcharts of some embodiments of the data reading process in the data management method of the storage system disclosed in the present invention are as follows: Figure 4 shown.
[0087] In step S41, the position of the data to be read and the length of the data to be read are obtained. Obtaining a data read request and determining the position of the data to be read and the length of the data to be read may adopt any method in the related art.
[0088] In step S42, the identifier of the first-level storage space to be read is determined according to the position of the data to be read and the length of the data to be read.
[0089] In some embodiments, similar to the data writing operation, when the length of the data to be read is less than or equal to the size of the first-level storage space, the position of the data to be read can be divided by the size of the first-level storage space to obtain the identifier of the first-level storage space to be read.
[0090] In some embodiments, when the length of data to be read is greater than the size of the first-level storage space, the reading operation involves at least two first-level storage spaces. The position of the data to be read is divided by the size of the first-level storage space to obtain the starting identifier of the first-level storage space to be read. The sum of the position of the data to be read and the length of the data to be read is then divided by the size of the first-level storage space to obtain the ending identifier of the first-level storage space to be read. The identifier of the first-level storage space to be read includes [start identifier, end identifier].
[0091] In step S43, data of at least one level of storage space in the corresponding storage space is read according to the identifier of the first level of storage space to be read.
[0092] In some embodiments, the level of the storage space to be read can be determined based on the target level information recorded during the data writing operation, thereby avoiding reading errors and improving reading accuracy.
[0093] In some embodiments, when the length of the data to be read is greater than the size of the first-level storage space, the data to be read stored in each first-level storage space is read separately, and then all the data to be read are aggregated.
[0094] For example, for the storage spaces corresponding to the start identifier and the end identifier, the level of the storage space to be read is determined according to the target level information recorded during the data write operation. In addition, if the end identifier - the start identifier > 1, then for each storage space corresponding to (start identifier, end identifier), the data of the storage space of the first level is read. The order of reading is summarized according to the storage order in the storage space to obtain all the data.
[0095] In some embodiments, each first-level storage space corresponds to a storage space header, e.g. Figure 2 As shown in , during the data reading process, the data of the storage space of the layer marked as valid is read according to the storage space header corresponding to the first-layer storage space to be read. For example, when the status information of the third-layer storage space includes an identification that valid information is stored, the data marked as valid information in the third-layer storage space is read; when the status information of the second-layer storage space includes an identification that valid information is stored, the data marked as valid information in the second-layer storage space is read; when the status information of the first-layer storage space includes an identification that valid information is stored, the data of the third-layer storage space is read. For the status information of the second-layer and third-layer storage spaces, if a bitmap is used, the data of the storage space corresponding to the position marked as valid in the bitmap is read.
[0096] The method in the above embodiment can improve the accuracy of data reading and avoid the occurrence of erroneous reading or missed reading.
[0097] In some embodiments, the reading priority of data in the third-level storage space is higher than the reading priority of data in the second-level storage space, and higher than the reading priority of data in the first-level storage space. For example, the data reading process adopts Figure 5 The method shown in .
[0098] In step 501, a storage space header chunk_head corresponding to an identifier of a first-level storage space to be read is obtained.
[0099] In step 502, the state information page_bit_map of the third-level storage space is processed to determine the position marked as valid, thereby determining the position of the third-level storage space to be read. Further, step 503 is executed.
[0100] In step 503, the state information block_bit_map of the second-level storage space is further read to determine the position marked as valid, thereby determining the data stored in the remaining block to be read in addition to the data in the page.
[0101] In step 504, the state information chunk_state of the first-level storage space is further read. If it is marked as valid, it is determined that the remaining chunks except the data in the page and block need to be further read.
[0102] In step 505, based on the data to be read determined in the above steps, the reading of data in the page, block, and chunk is completed.
[0103] Through this method, data can be read in the order of the third level, the second level, and the first level. Combined with the storage space header update method shown above, it can be concluded that this reading method can help improve the efficiency of data reading and reduce the probability of misreading and missed reading.
[0104] Based on the data management method in the embodiment shown above, through a multi-layer storage space data organization model, it is possible to select a suitable target level during the data reading process, reduce the amplification factor of small block read requests, and also reduce the number of write units involved in large block read requests, thereby improving reading efficiency.
[0105] By adopting the method in the above-mentioned embodiment of the present disclosure, when reading and writing large-block files, read and write operations can be performed at a larger-sized layer as a granularity, thereby improving the read and write bandwidth; when reading and writing small-block files, read and write operations can be performed at a smaller-sized layer as a granularity, thereby avoiding too much read and write amplification resulting in low IOPS (Input / Output Operations Per Second, the number of read and write operations per second), thereby improving the read and write efficiency.
[0106] Schematic diagrams of some embodiments of the data management device of the storage system disclosed in the present invention are as follows Figure 6 shown.
[0107] The data acquisition unit 61 can acquire the data to be written. In some embodiments, the data acquisition unit 61 can acquire the data to be written based on the method in the above step S11.
[0108] The identification determination unit 62 can determine the identification of the first-level storage space to be written according to the location of the data to be written and the length of the data to be written. In some embodiments, the identification determination unit 62 can execute the method of any embodiment shown in step S12 above.
[0109] The level determination unit 63 can determine the target level to which the storage space to which the data to be written is aligned belongs according to the size of the data to be written. In the embodiment of the present disclosure, the levels of the storage space are divided into the first level, the second level and the third level, and the target level is the first level, the second level or the third level. The storage space of the first level is larger than the storage space of the second level, for example, including at least two storage spaces of the second level; the storage space of the second level is larger than the storage space of the third level, for example, including at least two storage spaces of the third level. In some embodiments, the level determination unit 63 can execute the method of any embodiment shown in step S13 above.
[0110] The writing unit 64 can write the data to be written into the storage space of the target level corresponding to the identifier of the storage space of the first level.
[0111] Based on the device in the embodiment shown above, a multi-layer storage space data organization model is used to select a suitable target level during data writing, reduce the amplification factor of small block write requests, and also reduce the number of write units involved in large block write requests, thereby improving write efficiency.
[0112] In some embodiments, Figure 6 As shown, the data management device also includes an update unit 65, which can update the storage space header corresponding to the first-level storage space to be written. In some embodiments, each first-level storage space includes a corresponding storage space header, such as chunk_head. The storage space header stores the status information of the first-level storage space, such as whether the data currently stored in the first-level storage space is valid or invalid. The storage space header also stores the status information of the second-level storage space and the third-level storage space, which can be stored in the form of a bitmap, such as Figure 2 As shown in . Taking the first level as Chunk, the second level as Block, and the third level as Page as an example, the state information of the storage space of the first level is chunk_state, the state information of the storage space of the second level is block_bit_map, and the state information of the storage space of the third level is page_bit_map. In some embodiments, the updating unit 65 may execute the method in any of the above step S15 and steps 351-355.
[0113] Based on the data management device in the embodiment shown above, it is possible to identify the target level occupied by the data stored in the storage space through the storage space header, and update the occupancy status of the space at each level in the storage space, thereby facilitating the search for the correct data to be read when reading data, thereby improving the reliability of the reading operation.
[0114] In some embodiments, Figure 6 As shown, the data management device also includes a reading unit 66, which can obtain the reading position of the data to be read and the length of the data to be read. The identification determination unit 62 can also determine the identification of the first-level storage space to be read based on the reading position of the data to be read and the length of the data to be read. The reading unit 66 can also read data of at least one level of storage space in the corresponding storage space based on the identification of the first-level storage space to be read. In some embodiments, the reading unit 66 can execute the method in any of the embodiments in steps S41 and S43 above, and the identification determination unit 62 can also execute the method in any of the embodiments in step S42 above.
[0115] Based on the data management device in the embodiment shown above, through a multi-layer storage space data organization model, it is possible to select a suitable target level during the data reading process, reduce the amplification factor of small block read requests, and also reduce the number of write units involved in large block read requests, thereby improving reading efficiency.
[0116] A structural diagram of an embodiment of a data management device of a storage system disclosed in the present invention is shown in FIG. Figure 7 As shown. The data management device of the storage system includes a memory 701 and a processor 702. Wherein: the memory 701 can be a disk, a flash memory or any other non-volatile storage medium. The memory is used to store the instructions in the corresponding embodiment of the data management method of the storage system mentioned above. The processor 702 is coupled to the memory 701 and can be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The processor 702 is used to execute the instructions stored in the memory, which can improve the data writing efficiency of the distributed storage system.
[0117] In one embodiment, it is also possible to Figure 8 As shown, the data management device 800 of the storage system includes a memory 801 and a processor 802. The processor 802 is coupled to the memory 801 via a BUS bus 803. The data management device 800 of the storage system can also be connected to an external storage device 805 via a storage interface 804 to call external data, and can also be connected to a network or another computer system (not shown) via a network interface 806. No detailed description will be given here.
[0118] In this embodiment, the data writing efficiency of the distributed storage system can be improved by storing data instructions in a memory and processing the instructions through a processor.
[0119] In another embodiment, a computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the steps of the method in the corresponding embodiment of the data management method of the storage system. It should be understood by those skilled in the art that the embodiments of the present disclosure can be provided as methods, devices, or computer program products. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0120] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0121] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0123] So far, the present disclosure has been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.
[0124] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
[0125] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.
[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present disclosure rather than to limit it. Although the present disclosure has been described in detail with reference to the preferred embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present disclosure can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present disclosure, which should be included in the scope of the technical solution for protection of the present disclosure.
Claims
1. A data management method for a storage system, comprising: Get the data to be written; Determining an identifier of a first-level storage space to be written according to a writing position of the data to be written and a length of the data to be written; Determining a target level to which the storage space to which the data to be written is aligned belongs according to the size of the data to be written, wherein the target level is the first level, the second level, or the third level, the storage space of the first level includes at least two storage spaces of the second level, and the storage space of the second level includes at least two storage spaces of the third level; The data to be written is written into the storage space of the target level corresponding to the identifier of the storage space of the first level.
2. The data management method according to claim 1, further comprising: Update the storage space header corresponding to the first-level storage space to be written, Each first-level storage space includes a corresponding storage space header, in which the storage space header stores status information of the first-level storage space, status information of the second-level storage space, and status information of the third-level storage space.
3. The data management method according to claim 2, wherein: The storage status information of the first level is a validity identifier of the data stored in the storage space of the first level; and / or The state information of the second-level storage space is a bitmap of the second-level storage space storing valid information in the first-level storage space; and / or The status information of the third-level storage space is a bitmap of the third-level storage space storing valid information in the first-level storage space.
4. The data management method according to any one of claims 1 to 3, wherein: The step of determining the identifier of the first-level storage space to be written according to the writing position of the data to be written and the length of the data to be written includes: Determine a start mark according to a location where the data to be written is written and a size of each first-level storage space; The end identifier is determined based on the start identifier, the size of each first-level storage space and the length of the data to be written, wherein the identifier of the first-level storage space to be written includes the identifiers of all first-level storage spaces from the start identifier to the end identifier and including the start identifier and the end identifier.
5. The data management method according to claim 4, wherein: The step of determining the target level of the storage space to which the data to be written is aligned according to the size of the data to be written includes: Determine a starting position and an ending position of the data to be written in each of the first-level storage spaces according to a position where the data to be written is written, a length of the data to be written, and a size of each first-level storage space; According to the starting position and the ending position, and the sizes of the storage spaces of the first level, the second level, and the third level, the target level to which the storage space to be written to be aligned in each storage space of the first level belongs is determined.
6. The data management method according to claim 5, wherein: The determining of the starting position and the ending position stored in each of the first-level storage spaces to be written comprises: Determine the starting position and the ending position in the storage space corresponding to the start identifier according to the writing position of the data to be written and the size of each first-level storage space; According to the writing position of the data to be written, the length of the data to be written, the end identifier and the size of each first-level storage space, the end position in the storage space corresponding to the end identifier is determined.
7. The data management method according to claim 5 or 6, wherein: The determining of the starting position and the ending position stored in each of the first-level storage spaces to be written further comprises: when the difference between the ending identifier and the starting identifier is greater than 1, in the first-level storage space corresponding to the identifier between the starting identifier and the ending identifier, the starting position is 0, and the ending position is the size of the first-level storage space minus 1; The method of determining the target level to which the storage space to which the data to be written is aligned in each of the first-level storage spaces to be written belongs also includes: when the difference between the end identifier and the start identifier is greater than 1, in the first-level storage space corresponding to the identifier between the start identifier and the end identifier, the target level is the first level.
8. The data management method according to claim 3, wherein: The updating of the storage space header corresponding to the first-level storage space to be written includes at least one of the following: When the target level is the first level, updating the state information of the storage space of the first level to be valid, and updating the state information of the storage space of the second level and the third level to be invalid; When the target level is the second level, updating the location of the data to be written in the bitmap of the storage space of the second level to be valid, and updating the status information of the storage space of the third level to be invalid; When the target level is the third level, the location where the data to be written is stored in the bitmap of the storage space of the third level is updated to be valid.
9. The data management method according to claim 2, 3 or 8, further comprising: Get the position and length of the data to be read; Determining an identifier of a first-level storage space to be read according to the position of the data to be read and the length of the data to be read; According to the identifier of the first-level storage space to be read, data of at least one level of storage space in the corresponding storage space is read.
10. The data management method according to claim 9, wherein: In the reading of the corresponding storage space, data of at least one level of storage space includes: According to the storage space header corresponding to the first-level storage space to be read, data of the storage space of the level marked as valid is read.
11. The data management method according to claim 10, wherein: The step of reading data of the storage space of the level marked as valid according to the storage space header corresponding to the first-level storage space to be read includes: In a case where the state information of the storage space at the third level includes an identification of valid information stored, reading data identified as valid information in the storage space at the third level; When the state information of the storage space of the second level includes an identification of valid information stored, reading data identified as valid information in the storage space of the second level; When the state information of the storage space of the first level includes an identification that valid information is stored, data of the storage space of the third level is read.
12. The data management method according to claim 11, wherein: The reading priority of data in the third-level storage space is higher than the reading priority of data in the second-level storage space, and higher than the reading priority of data in the first-level storage space.
13. The data management method according to any one of claims 1 to 3, wherein: The first level is Chunk, the second level is Block, and the third level is Page.
14. A data management device for a storage system, comprising: A data acquisition unit, configured to acquire data to be written; an identification determining unit, configured to determine an identification of a first-level storage space to be written according to a writing position of the data to be written and a length of the data to be written; a level determination unit, configured to determine, according to the size of the data to be written, a target level to which the storage space to which the data to be written is aligned belongs, wherein the target level is the first level, the second level or the third level, the storage space of the first level includes at least two storage spaces of the second level, and the storage space of the second level includes at least two storage spaces of the third level; The writing unit is configured to write the data to be written into the storage space of the target level corresponding to the identifier of the storage space of the first level.
15. The data management device according to claim 14, further comprising: an updating unit, configured to update a storage space header corresponding to the first-level storage space to be written, Each first-level storage space includes a corresponding storage space header, in which the storage space header stores status information of the first-level storage space, status information of the second-level storage space, and status information of the third-level storage space.
16. The data management device according to claim 14, further comprising: a reading unit configured to obtain a reading position of the data to be read and a length of the data to be read, and read data of at least one level of storage space in the corresponding storage space according to an identifier of the first level of storage space to be read, The identification determination unit is further configured to determine the identification of the first-level storage space to be read according to the reading position of the data to be read and the length of the data to be read.
17. A data management device for a storage system, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the data management method according to any one of claims 1 to 13 based on instructions stored in the memory.
18. A computer-readable storage medium having computer instructions stored thereon, wherein the instructions, when executed by a processor, implement the data management method according to any one of claims 1 to 13.
19. A computer program product, comprising computer instructions, wherein when the computer instructions are executed by a processor, the data management method according to any one of claims 1 to 13 is implemented.